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The  contents  of  this  repoit  reflect  the  views  of  the  Human 
Resources  Research  Organization  which  is  responsible  foi  the  facts 
and  the  accuracy  of  the  data  presented  herein.  The  contents  do  not 
necessarily  reflect  the  official  views  or  policy  of  the  Department  of 
Transportation.  This  report  does  not  constitute  a standard,  specifi- 
cation, or  regulation. 


PREFACE 


I This  is  Volume  II  of  a two-volume  report  describing  the  analysis  of  the  Federal 

Aviation  Administration’s  Air  Navigation  Facilities  Maintenance  Training  Program. 
Volume  I consists  of  specific  objectives  of  the  study,  methods  of  procedure,  and  presents 
conclusions  pertaining  to  areas  where  the  training  program  can  be  improved.  The  preaoitt 
TA\*  ^volume  presents  the  recommendations  for  improvement  and  describes  basic  implementa- 
tion plans,  costs,  and  schedules  which  are  associated  with  the  recommended 
improvements^ 

The  worft'described  in  this  volume  of  the  report  was  performed  by  the  Human 
Resources  Research  Organization  (HumRRO),  Division  No.  1 (System  Operations), 
Alexandria,  Virginia*.  Dr.  J.  Daniel  Lyons,  Director,  under  the  sponsorship  of  the  Federal 
Aviation  Administratyon  (Contract  No.  DOT  FA  71  WA-2687).  Dr.  Alan  G.  Hundt  was 
Principal  Investigator,  Mr.  Robert  C.  Trexler  was  Program  Manager  and  Senior  Staff 
Scientist  assigned  to  the  project,  and  Mr.  P.J.  Butler,  Research  Scientist,  participated  fully 
in  the  preparation  of  the  recommendations  and  the  implementation  plans. 
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PROBLEM 


A comprehensive  examination  of  the  Air  Navigation  Facilities  (ANF)  Maintenance 
Training  Branch’s  program  was  undertaken  by  the  Human  Resources  Research  Organiza- 
tion (HumRRO)  under  an  award  by  the  Federal  Aviation  Administration  (FAA)  (Con- 
tract No.  DOT  FA71  WA-2687).  As  a consequence  of  the  study,  recommendations  were 
to  be  developed  as  well  as  implementation  plans,  possible  costs,  and  schedules. 


SUMMARY 

The  purpose  of  the  study  of  the  ANF  Maintenance  Training  Program  was  twofold. 
First,  a comprehensive  study  was  undertaken  to  determine  areas  where  the  program  could 
be  improved  in  its  effectiveness  and  in  its  costs.  Secondly,  recommendations  were 
constructed  based  on  the  study  and  a plan  was  devised  to  implement  the  recommenda- 
tions for  improvement  in  the  training  program. 


SPECIFIC  OBJECTIVES 

The  first  requirement  was  the  development  of  specific  recommendations  based  on 
the  findings  presented  in  Volume  I of  this  report.  This  was  performed  by  extracting  all 
finding  that  indicated  areas  for  improvement. 

Next,  the  recommendations  were  grouped  according  to  the  specificity  of  the  area 
recommended  for  improvement,  if  the  recommendation  pointed  to  a problem  which 
basically  was  indigenous  to  the  ANF  Training  Branch,  it  was  set  aside  for  presentation 
and  discussion  in  the  section  dealing  with  specific  recommendations  for  the  Branch  itself. 
If  it  was  determined  that  the  improvement  required  a revision  of  the  FAA’s  overall 
approach  to  ANF  maintenance  training,  the  recommendation  and  related  material  were 
set  aside  for  presentation  and  discussion  along  with  other  programmatic  recom- 
mendations. 

The  third  task  was  the  construction  of  plans  for  the  implementation  of  recom- 
mended changes  in  the  FAA  program  and  the  ANF  Branch’s  specific  activities.  Where  it 
was  possible  to  do  so,  approximations  of  the  amount  of  manpower  or  resources  necessary 
to  implement  the  recommendations  were  also  given.  Such  manpower  figures  and  costs 
should  be  taken  with  reservations  because  of  the  differences  in  costs  that  could  arise, 
depending  upon  decisions  related  to  the  implementation  of  the  recommendations,  and  the 
FAA’s  evaluation  of  the  scope  of  improvements  that  should  be  undertaken. 


METHODS  OF  PROCEDURE 

The  general  procedure  was  to  develop  the  recommendations  from  analysis  of  the 
comprehensive  study  performed  by  HumRRO  Secondly,  practical  principles  of  modem 
training  technology  and  education  wen-  utilized  to  define  solutions  to  many  of  the  areas 
requiring  improvement,  and  to  generate  the  estimates  of  implementation  costs.  All  efforts 
were  directed  toward  the  development  of  recommendations  and  plans  which  were 
specifically  suited  to  the  FAA’s  needs 


RECOMMENDATIONS 


T'^e  recommendations,  implementation  plans,  costs,  and  priorities  are  detailed  in  the 
following  sectii  ns.  In  general,  they  ful!  ir  to  two  major  categories : 

(1)  Recommendations  that  are  programmatic  and  require  a reorientation  of  the 
maintenance  training  program  toward  the  requirements  of  the  maintenance  task  itself. 
These  include  the  development  of  basic  documentation  which  defines  the  maintenance 
tasks  the  job  requirements,  job  performance  standards,  and  training  course  objectives.  A 
quality  control  program  is  an  essential  ingredient  of  the  training  and  operational  system. 
Considerable  improvements  in  the  present  quality  control  system  were  recognized  as 
necessary  and  specific  recommendations  for  developing  and  implementing  a quality 
control  system  were  presented. 

It  was  recommended  that  model  training  courses  be  developed  in  several 
content  areas,  along  with  documentation  and  procedures  designed  to  control  the  quality 
of  instruction.  It  was  emphasized  that  in-depth  involvement  of  FAA  Academy  and  other 
personnel  would  be  essential  in  the  development  of  model  programs  and  the  quality 
control  system.  Without  such  involvement,  the  utilization  of  the  models  and  the  develop- 
ment of  other  courses  by  FAA  personnel  would  lie  seriously  compromised. 

(2)  Recommendations  that  pirtain  to  areas  for  improvement  that  were  pri- 
marily restricted  to  activities  performed  within  the  ANF  Maintenance  Training  Branch 
lUeri.  While  many  of  tiiese  recommend itions  for  improvement  are  related  to  the  pro- 
grammatic changes  that  are  recommended,  it  is  still  true  in  most  cases  that  considerable 
improvement  tan  be  effected  in  some  areas  by  measures  that  can  be  taken  solely  by  ANF 
persornel.  Thi.-  is  not  intended  to  mear  that  all  of  the  problem  areas  which  have  been 
defined  exist  because  of  failures  in  the  ANF'  Branch  itself.  Rather,  it  is  a presentation  of 
steps  that  can  be  taken  at  the  local  level  to  improve  the  training  program,  which  are 
consist  mt  with  any  programmatic  changes  that  may  be  instituU*d.  For  the  most  part, 
they  can  proc  -ed  to  be  implemented  in  advance  of  the  programmatic  changes.  Conse- 
quent, immediate  improvement  in  the  program  can  be  realized  in  many  areas  by 
adopting  a number  of  important  corrective  measures. 

Because  the  present  contractor  was  required  to  present  costs  as  part  of  the 
implementation  p'an  estimates  were  given  where  appropriate.  This  occurs  where  it  was 
found  that  th<  FAA  has  options  on  how  they  may  wish  to  allocate  their  resources.  This 
hampers  primarily  in  areas  where  extensive  programmatic  changes  were  recommended.  In 
such  ii.starces,  manpower  requirements  were  estimated  for  the  performance  of  the  work 
by  non-F’AA  oigar.iza’ions  as  sjiecified  bv  the  contract.  The  use  of  non-FAA  personnel  to 
pe"form  si.ch  work  would  require  collaboration  with  FAA  personnel.  Consequently,  the 
implt  mentation  plan  describes  adjustments  that  should  be  made  in  the  event  the  recom- 
mendations are  not  directly  implemented  by  FAA. 

The  presentation  of  costs  and  adjustments  in  the  implementation  plans 
should  not  be  construed  as  a recommendation  that  the  work  should  be  performed  by 
noa  T \A  perso one..  The  information  is  presented  to  satisfy  a specific  contractual  require- 
ment When  . se  of  a contractor  is  recommended  as  an  alternative,  it  should  not  be 
construed  that  *.he  present  contractor  is  being  recommended. 
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Chapter  1 


PROGRAMMATIC  RECOMMENDATIONS 


The  major  programmatic  effort  the  FA  A should  undertake  to  improve  ANF 
technician  training  is  to  ensure  that  the  training  adequately  reflects  job  requirements. 

Two  related  activities  should  be  given  immediate  priority  to  achieve  this  objective. 
First,  job  requirements  need  to  be  determined.  Second,  training  programs  should  be 
constructed  to  take  into  account  job  requirements  information.  Other  facets  of  curric- 
ulum engineering1  should  be  developed  within  the  structure  of  the  ANF  Branch. 

Job  requirements  information  would  need  to  be  gathered  for  any  course  selected  for 
revision.  However,  because  of  the  effort  the  collection  of  such  data  entails,  it  would  not 
be  advisable  for  the  FAA  to  initiate  a large-scale,  wholesale  project  to  collect  it  for  all 
ANF  courses  at  once.  Consequently,  the  first  project  to  undertake  would  be  the 
determination  of  the  relative  priorities  among  the  courses  in  terms  of  revision.  Once  this 
listing  has  been  prepared,  the  collection  of  job  requirements  information  can  proceed  in  a 
rational  and  cost-effective  manner. 

The  development  of  an  in-house  capability  to  apply  modern  curriculum  engineering 
technology  can  be  undertaken  simultaneously  with  the  determination  of  priorities  for 
course  revision.  This  can  be  done  by  revising  one  course  in  each  of  the  five  career  fields 
within  the  ANF  Branch  to  provide  a model  for  the  courses  in  each  career  field. 
Participation  of  ANF  staff  in  the  development  process  would  result  in  their  in-depth 
appreciation  of  the  method  and  insure  their  ability  to  continue  with  the  process. 

Although,  in  the  interest  of  making  timely  improvements,  it  may  seem  desirable  to 
wait  until  course  revision  priorities  have  been  established,  it  is  inqierative  to  make  a start 
on  development  of  the  in-house  capability  immediately,  even  though  the  findings  of  the 
course  revision  priority  study  may  indicate  the  courses  under  revision  are  of  relatively 
low  priority.  It  is  recommended  that  the  courses  to  be  used  as  models  to  demonstrate  the 
methods  of  curriculum  engineering  be  selected  from  among  those  of  the  present  study,  in 
order  to  take  advantage  of  the  systematic  analyses  already  performed  on  them. 


PRIORITY  RANKING  OF  COURSES  FOR  REVISION 

Considering  the  number  of  courses  ANF  currently  conducts,  analytical  methods 
must  be  employed  to  identify  those  courses  which  should  undergo  revision.  First,  it 
would  be  wasteful  to  direct  the  ANF  Branch  to  simultaneously  conduct  job  analyses  and 
performance  standards  determinations  for  all  courses.  Some  courses  may  be  on  their  way 
to  extinction,  while  others  may  deal  with  equipments  that  have  a relatively  low  opera- 
tional criticality.  Consequently,  the  following  recommendation  is  made. 


'Curriculum  engineering.  as  used  here,  indicates  a systematic  and  orderly  process  for  the  deter- 
mination. design,  development,  and  evaluation  of  training  curricula  to  prepare  and  enable  personnel  to 
perform  their  assigned  jobs.  In  this  context,  curriculum  engineering  is  considered  to  be  a more  inclusive 
term  than  curriculum  development  in  that  the  engineering  of  a training  curriculum  involves  the  deter 
mination,  design,  and  evaluation  processes  as  well  as  the  development  process. 
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RECOMMENDED  ACTION 


Current  ANF  equipment-specific  training  programs  should  be  surveyed  to  determine 
the  relative  priority  of  revising  them. 

Two  products  would  result  from  the  survey:  (a)  the  list  of  courses  in  their  recom- 
mended order  for  revision,  and  (b)job  data  upon  which  to  base  the  selection  of  the 
means  for  performing  Job  Analyses. 

To  rank  order  courses  for  revision,  specific  criteria  must  be  established.  The  courses’ 
relative  priority  for  revision  should  certainly  take  cognizance  of  the  number  of  students 
involved  and  certain  cost  tradeoffs.  The  selection,  definition,  and  weighting  for  the 
criteria  should  be  determined  by  the  FAA  or  can  be  a joint  effort  of  a contractor  and 
FAA  officials.  Consideration  would  be  given  to  both  the  job  and  the  individual  training 
programs  required  to  meet  the  job  requirements. 

Certain  kinds  of  job  information  would  be  collected  for  use  in  establishing  criteria, 
for  example:  individual  training  programs  involved  and  their  sequence;  methods  used  in 
individual  training  programs;  length  and  cost  of  individual  training  programs,  and  student 
loads. 

The  length  of  the  training  programs  and  the  methods  involved  can  be  identified 
from  the  course  catalog.  Course  control  documents  with  their  related  Instructor  Guides 
would  be  needed  for  a more  detailed  study  of  the  programs  and  methods.  The  cost  of 
individual  training  programs  could  be  obtained  from  Training  Plans. 


IMPLEMENTATION 

This  study  should  commence  at  the  earliest  possible  date,  as  it  provides  a rational 
basis  for  ordering  the  course  revision  process. 

It  is  estimated  that  the  survey  would  require  approximately  six  man-months  of 
effort.  This  figure  could  vary  plus  or  minus  50%  depending  on  a more  precise  definition 
of  the  scope  of  effort.  It  would  result  in  a report  describing  the  methods,  findings,  results 
(the  ranking  in  time  sequence),  and  the  rationale  for  the  rankings. 


A MODEL  FOR  CURRICULUM  DEVELOPMENT 

One  way  for  the  FAA  to  acquire  additional  knowledge  of  effective  methods  of 
improving  their  instructural  system  would  be  for  curriculum  developers  to  obtain,  study, 
and  apply  the  guidance  presently  available  in  related  published  documentations.  For 
example,  the  U.S.  Army  Continental  Army  Command  (CONARC)  Regulation  350-100-1' 
provides  guidance  for  course  design,  and  The  Engineering  of  Educational  and  Training 
Systems 2 describes  methods  and  techniques  for  design  and  implementation  of  effective 
systems. 

As  an  alternative,  an  effective  way  to  transfer  the  necessary  skills  and  knowledges  to 
curriculum  developers  is  through  the  use  of  a model  or  demonstration  project.  Although 
it  would  be  possible  to  revise  an  existing  equipment  course  without  involving  course 
development  personnel,  it  would  not  be  wise  to  do  so.  First,  since  these  personnel  would 
not  have  participated  in  the  application  of  the  methods  or  the  construction  of  materials, 
they  would  not  have  the  experience  in  these  activities  or  the  appreciation  for  the  benefits 

1 U.S.  Continental  Army  Command  (CONARC).  Systems  Engineering  of  Training,  Regula- 
tion 350-100-1,  February  1968 

2 Robert  G.  Smith  The  Engineering  of  Educational  and  Training  Systems,  Lexington,  Mass.,  D.C. 
Heath,  1971 
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of  applying  the  model.  Second,  the  construction  of  an  <•!!>  ' 
not  only  the  application  of  knowledge  of  curriculum  • ■ngi 
expertise  in  the  technical  content  of  the  program  to  lx  ' r 

While  the  contractor  must  develop  some  level  of  oi*  r - 
expense  would  be  far  less  if  FAA  personnel  were  to  c <t*  » 1 

in  the  interests  of  timely  implementation  of  program  r*  » *■»  * 

desirable  to  make  use  of  the  content  expertise  already  >vaiiai> 

A joint  effort  among  the  contractor,  the  present  insiru  t>  »<ia 
organizational  elements  is  believed  to  be  the  most  cff*-.  uv«  1 i<:  ..«• 
employ  in  developing  the  capability  specifically  needed  • ■ 
specific  training  programs. 

While  the  methods  to  be  employed  are  quite  genera  . . * 

of  specific  training  problems  requires  considerable  judgmei  * 
that  differences  between  the  class  of  equipments  in  th*  <nm  • < 
the  class  of  equipments  in  the  data  processing  field  wi  • . 
approaches  to  the  application  of  the  methods.  In  additi  »m 
organized  for  the  conduct  and  administration  of  training  aJ.*ug  a* ■ • • ’ • 
appear  desirable  for  each  section  to  participate  in  a mud*- 
is,  one  course  from  each  of  the  five  career  fields  would  tx  « '*->! 

or  more  content  experts  from  each  of  the  presently  condu.  • * 

in  the  modeling  process. 

In  summary,  the  rationale  for  selecting  one  cours.  It  ■ • 

(a)as  a sample  they  provide  an  adequate  basis  for  

implementing  the  recommendations;  (b)  limiting  the  mn;  • 
would  afford  the  opportunity  for  a “faster  fix”  on  tin  ar>  • 

(c)  courses  from  each  career  field  would  be  necessary  to  funs  n 1 e-w  •-  . 

system  because  of  expected  differences  in  tiersonnel.  » quipn  • ■ 

requirements,  and  so  forth,  in  the  various  career  fields 

The  net  result  of  the  process  would  be  one  new  < '>ur-  - 

and  a core  of  experienced  curriculum  developers  in  ea>  t - • 

employed  to  administer  aid  participate  in  the  revision  of  add 
own  sections.  This  cadre  would  then  teach  others  how  t.  • o 
context  of  revising  other  courses.  In  this  way,  the  te.  hn< <)■  >*. . 

ANF  and  its  own  in-house  capability  bo  developed.  On  • u • •*  * * 

tional,  the  contractor’s  role  would  convert  into  a consulting 
that  the  curriculum  engineering  methods  are  effectively  ul...  ■ 1 ' "h 

course  development  on  a consulting  basis  would  be  appropn  it* 

RECOMMENDED  ACTION 

Development,  of  ANF  Branch  in-house  capability  to  *pp  • r • 

technology  should  he  undertaken  immediately  through  apph  if  • * 

“model”  course  from  each  of  the  five  career  fields. 


PLAN  DETAILS,  METHODS,  AND  PRODUCTS 

The  approach  to  he  used  would  be  to  re -engineer  a oh 
evaluated  in  this  study.  The  principal  advantage  of  this  appr  .a> 
would  he  required  because  of  the  familiarity  with  jul  . • »» ■ : - 


methodology  gained  as  the  result  of  the  work  performed  on  the  10  courses  and  presented 
in  Volume  I of  this  report. 

It  is  proposed  that  the  recommended  changes  be  implemented  and  the  training 
systems  engineering  process  be  demonstrated  in  the  following  courses  in  the  five  career 
areas: 


Course 


Career  Field 


ATCBI  - No.  40317 
Diesel  Engine  Generator  - No.  40103 
9020  CCC  for  Technician  - No.  43413 
Wilcox  Mark  I ILS  - No.  40216 
VHF/UHF  Direction  Finder  - No.  40210 


Radar 

Electromechanical 
Data  Processing 
Navigational  Aids 
Communications 


Assuming  these  courses  have  been  selected,  the  following  eight  steps  would  then  be 
applied  to  each. 


Step  1:  Select  a Set  of  Jobs  from  Each  Career  Field  for  Analysis 

Method:  With  the  selection  of  a particular  course  within  a career  field  (for  example 
ATCBI-3  in  the  Radar  field)  the  next  step  will  be  to  select  a set  of  jobs  within  that  field 
for  analysis.  Each  job  selected  would  involve  maintenance  responsibilities  for  the  equip- 
ment taught  in  the  course  selected.  However,  these  jobs  will  likely  involve  responsibilities 
for  maintenance  of  other  equipments  as  well.  The  jobs  selected  for  analysis  will  be 
representative  of  all  such  jobs  involving  maintenance  of  the  equipment  taught  in  the 
course  selected. 

Job  analysis  (Step  2)  requires  identification  of  the  specific  jobs  to  analyze.  Job  data 
in  Step  1 must  be  collected  in  order  to  identify  and  select  a sample  of  individual  jobs  for 
analysis  which  would  adequately  reflect  the  maintenance  requirements  of  the  equipment 
in  the  course  selected.  In  addition,  the  job  data  will  be  used  to  select  the  means  for 
performing  the  job  analysis. 

Step  2:  Perform  Job  Analysis 

Training  must  be  responsive  to  the  requirements  of  the  job.  A key  element  in 
constructing  a training  program  that  reflects  job  requirements  is  the  analysis  of  the  job 
itself. 

The  work  that  a technician  performs  in  a particular  job  is  determined  by  the  specific 
job  position.  If  a given  number  of  equipments  are  purchased  for  installation  at  opera- 
tional sites  which  have  the  same  design  and  configuration,  the  maintenance  tasks  imposed 
by  the  equipment  itself  should  be  the  same  with  respect  to  site  location. 

From  a technical  point  of  view,  the  quality  of  circuit  and  equipment  operation 
should  also  be  unchanged  from  site  to  site.  The  required  signal  levels  and  their  tolerances 
for  circuits  within  the  equipment  should  be  known  and  standard  for  all  systems  of  like 
type.  In  addition,  the  time  required  for  journeymen  technicians  to  perform  the  necessary 
checks,  adjustments,  alignments,  and  so  forth,  with  specification  of  accuracy  of  perform- 
ance, can  also  be  determined,  standard,  and  a joint  function  of  constraints  imposed  by 
the  hardware,  and  average  journeyman  capabilities. 

The  performance  quality  exhibited  by  the  journeyman  technician,  certified  to 
maintain  a given  equipment,  in  general,  exceeds  that  attained  in  formal  training.  That  is, 
journeyman  level  performance  is  only  fully  attained  after  some  period  of  performance  on 
the  job.  As  a consequence,  journeyman  standards  of  performance  must  be  determined. 
From  these,  entry  level  performance  standards  must  be  derived  which  can  then  be  used  as 
standards  to  direct  resident  training. 
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Method.  Based  upon  consideration  of  the  job  data  collected  in  Step  1,  a deter- 
mination would  be  made  of  the  most  economical  and  effective  means  for  collection  and 
compilation  of  the  job/task  data  required.  Then,  using  basic  job  and  task  analytical 
methods,  the  job  performances  required  and  standards  to  be  met  would  be  identified  and 
described.  This  effort  would  be  limited  to  maintenance  requirements  imposed  by  the 
specific  end  items  of  equipment  associated  with  the  job. 

Products 

(1)  An  inventory  of  the  maintenance  tasks  to  be  performed  and  the  minimum 
acceptable  standard  of  performance  for  each  task.  The  inventories  would  be 
organized  and  identifiable  separately  for  each  equipment  end  item  involved. 

(2)  Those  tasks  identified  as  requirements  for  the  training  program  selected 
from  each  job  would  be  described  in  detail.  The  description  would  provide  identifi- 
cation of  each  step  required  for  performance  of  the  task  in  terms  of  the  action  to 
be  taken,  the  object  to  be  manipulated,  and  the  means  for  determining  the  step  was 
performed  correctly.  Also  included  would  be  identification  of  significant  features  of 
the  work  environment  associated  with  task  performance— that  is,  tools  and  test 
equipment,  special  clothing  or  protective  apparel  required,  and  significant  environ- 
mental conditions  and  an  estimate  of  task  criticality. 

The  results  of  the  job  analysis  would  be  used  to  implement  Step  3. 

Step  3:  Select  Means  of  Training 

Method.  The  next  step  would  be  to  determine,  on  the  basis  of  the  individual  tasks, 
the  most  efficient  means,  or  combination  means,  of  training.  Means  of  training  considered 
would  include  resident,  formal  OJT,  and  directed  study.  The  selection  process  would 
involve  evaluation  of  the  tasks  against  the  following  criteria: 

(1)  Task  criticality 

(2)  Task  similarity  to  other  tasks  in  the  inventory 

(3)  Resource  requirements  and  availability  for  the  means  considered 

(4)  Relative  time  required  to  attain  proficiency 

(5)  Time  available  to  develop  proficiency 

(6)  Number  of  personnel  to  be  trained 

(7)  The  extent  to  which  field  maintenance  documentation  supports  and  guides 
task  performance 

(8)  Whether  a prerequisite  ability  for  task  performance  already  exists  in  trainee 
population 

(9)  Training  media  requirements  and  their  relative  appropriateness  to  the  means 
considered 

Products 

(1)  For  each  task  appearing  on  the  inventory,  the  identification  of  the  means 
of  training  to  he  used  and  the  level  of  proficiency  to  be  attained. 

(2)  Identification  of  the  terminal  behavioral  objectives  to  be  achieved  by  each 
means  of  training  selected. 

(3)  The  data  and  information  derived  from  the  above  analysis  would  serve  as 
the  data  base  for  Step  4. 

Step  4:  Perform  Analysis  of  I raining  Requirements 

Method.  Execution  of  this  step  would  involve  tour  sub-steps:  (a)  identification  of 
the  skills,  knowledges,  and  attitudes  requm'd  for  successful  task  performance;  (b)  con- 
version of  terminal  performance  objectives  to  training  course  objectives  and  criteria; 
(c)  development  of  the  coiure  structure;  and  (d)  development,  of  a concept  for  evaluating 
the  course.  The  detailed  task  data  and  work  environment  information  developed  under 


Step  2 would  be  analyzed  to  identify  the  knowledges,  skills,  and  attitudes  which  support 
task  performance.  The  identified  supporting  knowledges  and  skills  for  a task  and  the  job 
performance  standards  and  work  conditions  for  that  task  would  be  analyzed  and  con- 
verted into  training  course  objectives. 

Each  training  objective  would  specify  what  the  student  must  be  able  to  do,  the 
training  conditions  under  which  student  action  would  be  observed  and  the  acceptable 
level  of  performance  to  be  achieved.  A criterion  for  each  training  objective  would  be 
developed.  The  criterion  would  be  derived  from  the  training  objective  and  be  of  the  same 
format  and  type  of  information  contained  in  the  training  objective.  The  criterion  would 
state  the  student  performances  required  for  the  training  objective  to  be  successfully 
accomplished. 

Under  sub-step  3,  the  supporting  knowledges  and  skills  and  the  related  training 
objectives  would  be  analyzed  to  determine  an  effective  sequence  of  course  instruction. 
The  derivation  of  the  course  structure,  the  resulting  instruction,  the  sequencing  of 
material  would  be  developed  to  directly  reflect  the  performance  of  job  tasks.  The  final 
sub-step  would  involve  determination  of  the  most  appropriate  testing  schedule  for 
evaluation  of  student  achievement  of  course  objectives.  The  schedule  would  identify  the 
specific  test  points  within  the  general  sequence  of  course  instruction  and  specify  the 
course  content  upon  which  student  achievement  would  he  evaluated. 

Products 

(1)  The  specification  of  the  skills,  knowledges,  and  attitudes  to  be  developed 
during  training. 

(2)  A list  of  the  course  training  objectives  and  criterion  for  determining 
student  achievement  of  the  objectives. 

(3)  A basic  structure  of  the  training  course. 

(4)  A schedule  for  assessing  student  achievement  of  the  training  objectives. 

Step  5:  Develop  Training  Materials 

Method.  The  data  derived  from  the  training  analysis  (Step  4)  would  be  analyzed  to 
identify  specific  aspects  of  the  supporting  skills  and  knowledges  which  should  be  taught 
for  the  student  to  accomplish  the  training  objective.  This  analysis  and  the  final  selection 
of  course  content  would  be  based  upon  consideration  of  the  characteristics  and  abilities 
of  the  trainees  and  personnel  to  serve  as  instructors. 

The  instructional  aspects  or  points  would  then  be  examined  to  derive  and 
establish  an  effective  sequence  of  instruction.  Pertinent  information  source  and  reference 
materials  would  be  identified  for  each  instructional  point.  These  materials  would  be  used 
in  developing  most  of  the  total  content  for  the  instructional  points. 

The  selection  of  the  method  of  instruction  to  be  used  would  be  based  upon 
consideration  of  what  the  student  must  do  to  satisfy  the  training  objective  and  the 
relative  cost  and  effectiveness  of  alternative  methods  for  facilitating  the  instruction  to  be 
accomplished.  The  method  or  combination  of  methods  selected  would  be  identified  for 
each  instructional  point.  The  methods  of  instruction  selected  and  the  instruction  to  be 
presented  would  be  examined  to  identify  the  training  media,  aids,  and  equipment 
required  for  the  course.  The  selection  process  would  give  consideration  to  their  relative 
effectiveness  in  supporting  thy  method  of  instruction,  their  relative  cost,  and  the 
demands  they  would  place  upon  the  existing  training  facilities. 

After  estimating  the  amount  of  time  required  for  the  instructional  points,  the 
training  objectives  would  be  grouped  into  lesson  units  and  the  time  required  to  present 
each  lesson  would  be  estimated.  The  previously  derived  information  on  each  lesson  would 
be  used  to  develop  a lesson  plan  or  guide  of  a standardized  format  and  content.  Other 
course  control  documents  to  be  developed  would  include  a program  of  instruction  (POl) 
and  training  schedule. 
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Requirements  for  training  manuals,  student  handouts,  and  other  types  of 
training  literature  would  be  identified.  The  design  and  content  of  these  materials  would 
lie  selected  to  provide  maximum  support  to  student  learning  and  achievement  of  the 
course  training  objectives. 

Products 

(1)  Course  lesson  plans,  program  of  instruction  and  training  schedule. 

(2)  Student  handout  materials. 

(3)  Training  manuals  and  similar  literature. 

(4)  Training  media  and  aids  requirements. 

(5)  Training  equipment  or  simulator  requirements. 

Step  6:  Develop  Course  Testing  Materials 

The  tentative  testing  schedule  developed  in  Step  4 would  provide  the  basic  frame- 
work for  this  development  effort  Each  training  objective  would  be  examined  to  identify 
the  specific  aspects  to  be  tested.  Alternate  versions  of  the  training  objectives  would  be 
identified  for  testing  purposes  and  a plan  for  sampling  as  many  of  these  variations  as 
possible  would  be  developed.  A training  objective  concerning  troubleshooting,  for 
example,  could  have  numerous  different  versions  for  testing  purposes. 

The  basic  purpose  of  the  tests  to  be  developed  would  be  the  evaluation  of  student 
achievement  of  the  specified  training  objectives.  They  would  be  performance  oriented  and 
directed  toward  assessment  of  the  total  task  performance  rather  than  individual  task 
components. 

The  test  construction  effort  would  place  heavy  emphasis  upon  test  realism  in  terms 
of  job  requirements  in  the  field,  objectivity  in  test  administration  and  scoring,  compre- 
hensiveness in  coverage  of  all  training  objectives  and  the  utility  of  testing  results  for 
controlling  the  quality  of  training. 

The  job  performance  requirements  from  Step  2 and  the  training  objectives  would  be 
analyzed  to  identify  specific  test  program  standards.  These  standards  would  establish  the 
minimum  standard  of  student  performance  on  each  training  objective  which  would  be 
acceptable  when  measured  by  the  testing  program.  Sub-standard  performance,  as  reflected 
by  the  testing  program,  would  indicate  the  need  for  improvement  in  the  quality  of  the 
training.  Each  standard  would  specify  the  task  to  he  performed,  the  conditions  under 
which  the  student  should  perform  the  task,  and  the  number  of  correct  responses 
required.  In  addition,  written  procedures  would  be  prepared  for  the  administration  and 
scoring  of  the  tests  develop'd. 

The  final  step  in  development  of  testing  materials  would  be  to  check  the  validity  of 
the  tests  and  to  identify  difficulties  associated  with  test  administration  from  the  view- 
point of  both  the  instructor  and  student. 

Throughout  this  development  effort,  consideration  will  be  given  to  the  time,  effort, 
and  resources  requited  to  accomplish  the  testing  proposed  and  suitable  compromises  will 
be  identified  and  examined. 

Products 

(1)  A complete  set  of  testing  materials  to  include  alternate  versions  of  the 
same  test  or  test  item. 

(2)  Test  standards. 

(3)  A testing  schedule. 

(4)  Written  procedures  for  administration  and  scoring  of  tests. 

(5)  Identification  of  the  test  or  test  items  to  he  used  in  measuring  achievement 
of  ea<  h training  objective. 
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Step  7:  Develop  Training  Quality  Control  System 

Method.  In  terms  of  scheduling  the  various  steps  in  the  system  engineering  process, 
this  step  can  he  initiated  shortly  after  Step  5 is  completed,  and  can  be  executed 
concurrently  with  Step  6.  This  would  he  possible  because  different  groups  of  personnel 
will  be  involved  in  Steps  t>  and  7. 

The  prime  purpose  of  the  quality  control  system  will  be  to  continually  assure 
that  the  product  of  training  (the  trainee)  can  meet  the  entry  level  requirements  of  the 
job  and  that  the  means  employed  by  the  training  system  in  producing  this  product  are 
cost-effective.  Organizationally,  (he  quality  control  staff  should  be  free  from  direct 
supervision  from  those  organizational  elements  responsible  for  the  development,  conduct, 
and  administration  of  the  training  to  be  controlled. 

Inasmuch  as  a number  of  the  elements  in  a quality  control  system  are  known 
to  presently  exist  at  the  Academy,  discussion  of  the  execution  of  this  step  will  be  limited 
to  those  quality  control  features  pertinent  to  the  recommendations  of  this  study  and 
found  to  be  absent. 

The  effectiveness  of  any  training  quality  control  system  is  dependent  upon  the 
extent  to  which  training  adjustments  and  changes  are  based  upon  empirical  data  rather 
than  opinions  or  judgments.  The  proposed  implementation  plan  calls  for  the  development 
of  tests  and  procedures  for  a more  objective  evaluation  of  the  effectiveness  and  quality  of 
training  programs.  The  first  type  of  test  to  be  developed  would  be  one  for  monitoring 
the  effectiveness  of  training.  The  key  characteristics  of  these  tests  will  be  that  (a)  they 
will  measure  the  student’s  ability  and  proficiency  on  total  tasks,  (b)  they  will  be  based 
upon  and  reflect  course  training  objectives,  and  (c)  they  will  be  scored  on  a pass-fail 
basis.  The  emphasis  will  be  on  measurement  of  what  the  student  is  able  to  do. 

A second  type  of  tests  will  be  developed  to  identify  and  determine  the  cause  of 
training  ineffectiveness.  They  would  be  diagnostic  in  purpose  and  would  be  used  to 
determine  why  students  failed  to  meet  the  criterion  for  given  training  objectives.  They 
would  measure  the  student’s  ability  on  the  skills  and  knowledge  components  of  a task 
and  provide  data  for  identifying  the  instructional  content  contributing  to  the  observed 
failure.  An  effort  would  be  made  to  check  the  reliability  and  validity  of  the  tests 
developed.  Procedures  will  be  prepared  for  the  administration  and  scoring  of  these  tests 
and  the  interpretation  of  results. 

Products 

(1)  Testing  materials  and  procedures  for  monitoring  the  quality  of  training. 

(2)  Testing  materials  and  procedures  for  diagnosis  of  instructional  difficulties 
and  instances  of  training  ineffectiveness. 

(3)  Procedures  to  provide  a continual  flow  of  job-oriented  performance  data 
upon  which  to  base  decisions  for  the  improvement  of  training  quality  and 
effectiveness. 

Step  8:  Conduct,  Evaluate,  and  Revise  Re  engineered  Training  Courses 

Meth od.  The  actual  conduct  of  the  training  would  be  accomplished  by  the  ANF 
instructional  staff.  However,  all  changes  in  training  media,  techniques,  and  methodology 
would  be  discussed  by  the  contractor  with  course  supervisory  and  instructor  personnel. 
The  purpose  of  the  discussions  would  be  to  describe  the  rationale  for  the  change  and 
identify  the  requirements  for  successful  implementation. 

The  evaluation  effort  would  be  accomplished  by  Academy  personnel  and  would 
collect  data  to  reflect  upon  two  aspects  of  the  revised  course;  (a)  the  effectiveness  of  the 
course  in  meeting  its  stated  objectives  and  in  producing  personnel  who  can  satisfy 
requirements  of  the  job;  and  (b)  the  nature  and  extent  of  improvements  realized  from  the 
implementation  of  recommended  changes.  Collaborative  effort  would  be  required  to 
effectively  specify  the  means  of  evaluation  and  criteria. 
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The  results  of  the  evaluation  would  he  examined  jointly  to  seek  the  means  and 
methods  for  correcting  identified  course  deficiencies  and  gaining  additional  improvements. 


IMPLEMENTATION  OF  CHANGES 

Active  participation  of  ANF  personnel  will  be  required  to  achieve  the  goal  of 
transferring  the  technology  of  curriculum  development  through  the  model  restructuring 
of  existing  training  programs.  Although  it  is  desirable  to  revise  five  courses  simul- 
taneously, it  would  be  more  efficient  from  a program  point  of  view  to  phase  the 
initiation  of  the  particular  course  revision  programs  in  time.  The  reason  for  this  recom- 
mendation is  that  working  relationships  between  any  chosen  contractor  and  ANF 
personnel  must  be  developed,  and  administrative  arrangements  must  be  made  for  the 
allocation  and  execution  of  the  required  work.  It  seems  wise  to  establish  these  arrange- 
ments while  performing  the  work  required  for  a single  course  rather  than  simultaneously 
for  all  five  courses.  This  would  take  advantage  of  the  possibility  that  the  solutions 
derived  for  the  first  application  might  work  for  other  courses,  resulting  in  considerable 
savings. 

It  is  recommended  that  Course  No  40210  - VHF/UHF  DF  be  selected  as  the  initial 
course  for  revision,  not  because  it  is  in  any  greater  need  than  any  other,  but  because  it  is 
relatively  short  (two  weeks)  and  deals  with  an  equipment  that  has  no  known  modifica- 
tions. Thus,  the  model  could  be  developed  quickly. 

The  following  comments  are  indicative  of  the  degree  of  involvement  expected  of 
ANF  personnel  in  support  of  the  revision  effort  as  it  is  currently  presented. 

Step  1:  Select  a Set  of  Jobs  From  Each  Career  Field  for  Analysis.  The  job  would 
be  that  required  for  the  maintenance  of  the  VHF/UHF  DF.  The  objective  of  the  task 
would  be  to  acquire  job  information  in  order  to  select  the  most  feasible  method  of 
performing  the  job  analysis.  This  task  would  be  executed  by  the  contractor  in  consulta- 
tion with  a variety  of  FAA  organizational  elements.  Once  the  means  have  been  selected 
to  perform  the  job  analysis,  the  ANF  instructor  assigned  to  the  project  would  begin  his 
participation. 

Step  2:  Perform  Job  Analysis.  Depending  on  the  means  selected,  the  instructor 
might  be  requested  to  draft  a job  analysis  survey  questionnaire;  review  and  critique  a 
questionnaire  prepared  by  the  contractor;  visit  selected  operational  sites  to  interview  job 
holders  in  the  company  of  the  contractor  for  the  purpose  of  developing,  or  trying  out 
the  questionnaire,  or  to  collect  job  requirements  information. 

The  instructor  would  be  expected  to  organize  the  maintenance  tasks  identified  into 
an  inventory  in  a format  prescribed  by  the  contractor  according  to  instructions  provided. 
He  would  also  prepare  descriptions  of  each  task  identified  as  a training  program  require- 
ment, collaboratively  and  interactively  with  contractor  personnel. 

Step  3:  Select  Means  of  Training.  The  assigned  instructor  would  then  participate  in 
the  determination  of  the  most  efficient  means  or  combination  of  means  of  training  and 
the  level  of  proficiency  to  be  attained  in  the  individual  tasks,  according  to  specific 
criteria 

Step  4:  Perform  Analysis  of  Training  Requirements.  In  this  step  the  instructor 
would  learn  how  to  identify  the  skills,  knowledges,  and  attitudes  required  for  successful 
task  performance,  to  convert  terminal  performance  objectives  to  training  course  objectives 
and  criteria,  and  to  develop  the  course  structure  and  would  participate  in  the  develop- 
ment of  a concept  for  evaluating  the  course.  Using  the  data  base  obtained  earlier,  the 
contractor  would  work  closely  with  the  instructor  to  ensure  that  he  knows  how  to 
perform  the  required  work. 
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Table  1 


Man  Days  Required  to  Perform  Work  Indicated  for 
Steps  1 Through  8 for  Course  No.  40210 


Step 

Task 

Estimated  Man- 
Days  Required 

1 

Select  Job  Analysis  Means 

20* 

2 

Perform  Job  Analysis 

40 

3 

Select  Means  of  Training 

10 

4 

Perform  Analysis  of  Tiaining  Requirements 

20 

5 

Develop  Training  Materials 

40 

6 

Develop  Testing  Materials 

25 

7 

Develop  Quality  Control  System 

35 

8 

Conduct,  Evaluate  and  Revise  Program 

35 

225  man-daysb 

aThis  figure  also  includes  time  for  program  definition  and  start  up. 
Estimated  man-days  required  exclusive  of  FAA  personnel  participation. 


maintenance  functions  at  specific  levels  of  performance  when  they  report  for  duty  at  a 
field  site.  It  is  normally  expected  that  training  programs  which  utilize  prime  equipment  at 
the  Academy  will  provide  the  trainee  with  the  basic  information,  skills,  and  procedures 
that  he  needs  to  further  develop  his  competence  as  an  ANF  technician.  How  does  the 
procurement  process  actually  work?  Is  the  prime  equipment  always  necessary  to  achieve 
these  goals?  What  would  be  an  effective  procurement  policy  for  the  ANF  Training 
Program? 

Examination  of  the  present  method  for  processing  equipment  has  shown  that  it  is  an 
informal  process.  Although  the  data  that  were  collected  showed  that  substantial  errors  in 
procurement  of  equipment  for  training  purposes  have  not  occurred,  the  equipment 
problem  grows  increasingly  more  complex.  A more  fully  documented  equipment  procure- 
ment procedure  should  tie  established. 

The  solution  to  the  procurement  problem  depends  as  much  upon  exercising  basic 
principles  of  modem  technology  as  it  does  upon  utilizing  an  artful  and  reasonable 
approach  to  the  problem. 

The  definition  of  an  equipment  procurement  policy  requires  three  principal  com- 
ponents: (ala  basic  logic  or  rationale  for  procurement;  (b)  the  definition  and  availability 
of  certain  kinds  of  data  which  pertain  to  each  specific  procurement  problem;  (c)  the 
availability  of  knowledgeable  and  responsible  people  to  activate  the  decision-making 
machinery. 

A system  employed  by  the  U.S.  Air  Force  since  the  early  1960s,  described  in  Air 
Force  Systems  Command  (AFSC)  Manual  80-3.'  obtains  data  from  manufacturers  during 
the  early  conceptualization  and  development  phases  of  new  equipment  acquisition.  Such 
data  are  used  to  develop  a Training  Equipment  Planning  Information  report  describing 
training  equipment  requirements  well  in  advance  of  production  of  the  equipment. 

1 Air  Force  Systems  Command  Handbook  of  Instruction  for  Aerospace  Personnel  Sub-Systems 
Design,  AFSC  Manual  HO  3.  January  10(56 
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While  present  training  technology  does  allow  the  computation  of  equipment  costs 
amortized  over  specified  benefit  periods,  effectiveness  measures,  and  precise  calculation  of 
the  answers  to  some  problems,  the  basic  procurement  question  and  its  answer  are  most 
usefully  described  in  extremely  straightforward  terms. 

Step  1.  Early  in  the  development  of  a procurement  of  operational  equipment,  the 
manufacturer  and  the  Federal  Aviation  Administration  should  ensure  that  a detailed 
specification  of  the  maintenance  requirements  for  the  device  are  defined.  Considerable 
care  should  be  taken  to  ensure  that  the  requirements  are  specific,  that  they  define  the 
specific  maintenance  responsibilities  of  the  individual  technician,  the  levels  of  skills, 
amounts  of  knowledge,  and  procedures  that  the  technician  must  have  at  his  disposal  to 
maintain  the  equipment  effectively. 

Step  2.  Basic  information  developed  during  Step  1 should  then  be  reexamined  to 
determine  the  minimum  job  entry  level  performance  standards  that  the  system  mainte- 
nance service  will  accept  from  a technician  reporting  on  site  for  duty  to  maintain  the 
new  operational  equipment.  This  re-examination  of  maintenance  requirements  is  necessary 
in  light  of  the  very  strong  possibility  that  the  field  site  may  seldom  require  that  the 
technician  reporting  for  duty  be  able  to  perform  maintenance  at  the  same  level  as  a more 
experienced  technician.  Therefore,  the  standards  applied  to  the  new  technician  may  vary 
from  those  that  he  would  be  expected  to  meet  after  a period  of  time  on  the  job. 

The  job  entry  level  performance  standards  should  be  specified  in  terms  of: 
(a)  specific  skills  which  the  technician  must  have  and  which,  necessarily,  require  practice 
on  operational  equipment;  (b)  maintenance  procedures  which  he  is  to  follow  which  do 
not  require  practice,  but  which  are  guided  by  job  aids  and  ancillary  information; 
(c)  classroom  knowledge  which  provides  sufficient  background  for  the  technician  to 
implement  troubleshooting  procedures,  utilize  job  aids,  and  understand  the  application  of 
maintenance  skills. 

Step  3.  Specific  training  objectives  and  standards  should  be  established  based  on  the 
inventory  of  acceptable  job  entry  level  of  performance  standards.  Training  objectives  or 
the  specific  training  outcomes  to  be  required  would  be  basically  the  skill,  procedure,  and 
knowledge  capabilities  expected  of  the  technician  when  he  reports  for  duty  on  the  site. 
That  is,  the  training  outcomes  required  or  the  training  objectives  should  be  nearly 
identical  with  the  job  entry  level  skill,  procedures,  and  knowledge  standards. 

Step  4.  The  skill,  knowledge,  and  procedures  training  outcomes  or  requirements 
which  have  been  defined  must  then  be  compared  with  skill,  knowledge,  and  procedures 
training  outcomes  presently  established  as  objectives  in  existing  courses  in  the  ANF 
maintenance  training  program.  This  would  require,  for  each  procurement,  a detailed 
examination  of  the  maintenance  requirements  of  the  new  operational  equipment  and  its 
associated  training  requirements  with  the  training  requirements  of  existing  courses. 

If  the  examination  were  to  reveal  that  maintenance  of  the  new  operational  equip- 
ment does  not  require  that  a new  skill  is  required  for  the  technician  to  meet  the  job 
entry  level  standards,  then  there  is  seldom  any  justification  for  the  addition  of  the 
operational  equipment  to  the  Academy’s  inventory.  Instead,  the  additional  knowledge 
and  procedures  that  the  technician  should  know  which  are  peculiar  to  the  equipment  can 
be  imparted  through  standard  classroom  procedures  utilizing  inexpensive  mock-ups,  films, 
photographs,  and  other  inexpensive  audio-visual  materials. 

If  at  least  one  new  skill  must  be  imparled  to  the  technician,  the  nature  of  that  skill 
must  be  examined.  If  the  development  of  the  skill  requires  that  the  entire  operational 
system  be  available  for  troubleshooting  practice,  then  the  equipment  should  be  procured 
for  use  at  the  Academy  on  either  a long  or  short  term  basis,  depending  upon  the  training 
pipeline  requirement.  If  development  of  the  skill  does  not  require  that  the  entire 
operational  system  be  available  for  troubleshoot  ing  practice,  then  a feasibility  study 
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should  be  undertaken  to  determine  whether  a part  task  trainer1  can  be  procured  at  a cost 
which  is  substantially  lower  than  that  of  the  operational  equipment.  Since  operational 
equipment  located  at  the  Academy  for  training  purposes  frequently  is  sent  to  the  field 
after  its  training  mission  has  been  fulfilled,  it  would  be  unlikely,  generally,  that  the 
procurement  of  a part  task  trainer  would  be  cost  beneficial.  However,  each  case  should 
he  examined  on  its  own  merits. 


IMPLEMENTATION  OF  THE  PROCUREMENT  POLICY 

Should  this  detailed  analysis  be  performed  each  time  operational  equipment  is  being 
procured?  Is  it  really  practical  to  spend  resources  to  examine  each  procurement  and  to 
what  level  of  detail  should  each  procurement  be  examined? 

The  cost  of  the  analysis  that  has  been  described  might  appear  to  be  prohibitive. 
However,  it  should  be  recognized  that  nearly  all  of  the  information  that  has  been 
outlined  as  necessary  to  implement  the  decision-making  process  is  the  same  information 
that  has  been  recommended  as  being  vital  for  the  specification  of  maintenance  training 
courses.  It  has  been  previously  recommended  that  Maintenance  Concepts,  Job  Analyses, 
and  related  documentation  must  be  generated  in  order  to  increase  the  cost-effectiveness 
of  developing  maintenance  training  courses  or  programs.  Other  than  the  additional 
requirement  to  examine  the  functional  capabilities  of  existing  equipment  to  see  if  they 
satisfy  new  requirements,  no  additional  information,  or  information  burden,  would  be 
imposed  upon  the  FAA  to  provide  the  basis  for  its  procurement  decision  making. 

The  final,  basic  consideration  is  timeliness.  That  is,  the  development  of  the  Mainte- 
nance Concepts,  job  analyses,  job  entry  level  performance  standards,  specific  training 
objectives  and  standards,  must  all  begin  in  the  conceptual  stage  of  the  procurement  of 
the  equipment.  The  information  must  be  developed  and  refined  as  the  procurement 
process  itself  proceeds  and  is  refined. 

The  cost  of  implementing  the  procurement  decision  process  is  therefore  shared  with 
the  cost  of  basic  course  definition.  Actually,  this  is  where  the  real  savings  in  training 
resources  can  occur.  The  comparison  of  job  skills,  knowledges,  and  so  forth,  required  by 
the  new  maintenance  requirement  with  the  skills,  knowledges,  and  procedures  which  are 
already  the  subject  of  instruction  at  the  Academy  is  basic.  The  comparison  must  be 
performed  to  determine  whether  or  not  courses  should  be  established,  their  curriculum 
content,  and  ultimately  the  types  of  training  devices  which  must  be  available  to  imple- 
ment the  training  requirements. 

The  possibly  erroneous  decision  to  expend  $100,000  to  $200,000  for  operational 
equipment  for  training  purposes  is  of  lesser  significance  than  an  erroneous  decision  to 
develop  and  maintain  courses  which  need  not  be  developed  or  which  may  be  unneces- 
sarily long.  The  costs  of  conducting  unnecessary  training  in  the  long  run  can  far  exceed 
the  cost  of  an  operational  equipment  for  training. 

Consequently,  it  is  recommended  that  this  assessment  be  made  part  of  the  integrated 
study  of  the  need  for  any  new  course  and  for  the  specification  of  its  curriculum. 


1 A part-task  trainer  is  “a  device  which  permits  selected  aspects  of  a task  to  be  practiced  inde- 
pendently of  other  elements  of  the  task.  Its  purpose  is  to  provide  economical  training  on  certain  elements 
requiring  special  practice  which  are  not  dependent  on  the  total  equipment;  for  example,  the  Malfunction 
Circuitry  Trainer  (MAC)."  AFSCM  80  .1  Handbook  of  Instruction  for  Aerospace  Personnel  Subsystems 
Design. 
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QUALITY  CONTROL  OF  TRAINING 


The  requirement  for  an  improved  Quality  Control  System  has  already  been  discussed 
in  the  context  of  curriculum  development  and  elsewhere  in  the  report.  The  purpose  of 
this  section  is  to  present  the  general  rationale  for  the  establishment  of  an  improved 
system,  and,  secondly,  to  describe  a system  that  is  appropriate  to  ANF  Maintenance 
Training. 

Rationale 

It  has  been  repeatedly  stated  that  training  programs  must  be  tied  to  specific 
performance  requirements  with  measurable  standards.  However,  even  when  such  programs 
have  been  constructed  and  are  in  use,  they  require  periodic  monitoring  and  adjustment. 
The  most  carefully  engineered  curriculum  carries  with  it  no  guarantee  of  perfection.  It  is 
to  be  expected  that  the  initial  conduct  of  a course  or  program  will  reveal  inadequacies. 
Some  of  these  may  be  related  to  the  students’  inabilities  to  demonstrate  mastery  of  tasks 
at  specified  training  levels.  Attention  must  be  directed  to  determining  why  this  has 
occurred,  and  corrective  action  must  be  taken  to  prevent  its  recurrence.  The  solution 
might  be  quite  simple:  Students  should  be  given  more  time  to  develop  the  skill  being 
tested,  or,  on  further  study,  it  might  be  found  that  insufficient  time  was  used  in 
developing  the  skills  required  for  the  execution  of  a related  subtask. 

Because  the  quality  of  the  product— a trained  technician— needs  to  be  maintained, 
the  quality  of  the  training  process  needs  to  be  controlled.  The  means  by  which  the 
quality  of  the  process  is  measured  is  through  measuring  the  quality  of  the  product.  If  the 
trainee  cannot  perform  the  task,  the  Quality  Control  System  must  examine  those  parts  of 
the  program  that  have  been  designed  to  teach  him  that  task,  and  recommend  changes 
based  on  additional  diagnostic  measurements. 

It  is  also  true  that  task  mastery  may  be  achieved  at  needless  expense.  The  Quality 
Control  System  must  examine  not  only  situations  in  which  trainees  fail  to  perform  to 
criteria,  but  also  those  in  which  trainees  always  master  the  task.  While  task  mastery  is  the 
goal,  closer  examination  of  the  process  may  show  that  task  mastery  can  be  achieved  in 
less  time  than  presently  devoted  to  it.  Thus,  another  function  the  Quality  Control  System 
can  serve  is  to  ensure  that  training  is  accomplished  economically.  The  use  of  the  system 
can  be  very  effective  in  evaluating  any  proposal  to  modify  a training  course  or  program. 
If  the  rationale  behind  the  proposed  modification  is  a cost/beneficial  improvement  in  a 
training  course,  quality  control  data  on  the  current  effectiveness  of  the  course  should  be 
a part  of  the  evidence  supporting  the  recommended  modification. 

It  is  a general  principle  that  the  quality  control  function  should  be  performed  by 
personnel  who  are  not  involved  in  the  process  of  producing  the  item  being  examined. 
Objective  measurements  and  conclusions  are  more  easily  obtained  by  independent 
“inspectors.” 

The  Evaluation  and  Standards  staff  (AC980)  presently  has  a quality  control  function 
at  the  Academy.  However,  it  is  currently  understaffed  to  assume  the  quality  control 
activities  required  to  effectively  monitor  and  evaluate  training  programs.  Consequently, 
were  the  Evaluation  and  Standards  staff  to  be  assigned  the  function  of  conducting 
diagnostic  testing  to  determine  where  improvements  need  to  be  made,  and  to  examine 
the  effectiveness  of  improvements,  its  size  should  be  increased  considerably. 

It  is  recommended  that  the  FAA  develop  an  improved  Quality  Control  System. 
Figure  1 illustrates  the  essential  components  of  the  system.  It  must  be  noted  that  the 
system  crosses  many  organizational  lines  and  spans  many  areas  of  responsibility. 


16 


A 


Data  Flow  Diagram  Quality  Control  System 


Figure  1 


The  system  would  be  constructed  and  function  in  the  following  manner: 

(1)  SMS,  in  cooperation  with  TR1,  determines  both  the  job  performance 
requirements  and  job  entry  level  performance  requirements. 

(2)  TR1  develops  the  training  objectives  and  performance  requirements  and 

transmits  them  to  the  Academy  and  to  the  Evaluation  and  Standards  staff  for  each 
course.  ® 

(3)  SMS  transmits  job  performance  requirements  and  job  entry  level  per- 
formance requirements  to  the  field  where  trainees  will  ultimately  be  assigned,  and  to 
the  Evaluation  and  Standards  staff. 

(4)  The  Academy  transmits  the  training  objectives  and  performance  require- 
ments information  to  the  academic  personnel  responsible  for  course  construction  for 
their  use. 

(5)  After  the  course  has  been  conducted,  trainees  are  returned  to  their  field 

sites. 

(6)  Using  the  information  supplie-l  by  SMS  on  job  performance  requirements, 
and  job  entry  level  performance  requirements,  field  personnel  evaluate  the  trainee 
after  a short  period  on  the  job,  and  data  are  returned  to  the  Evaluation  and 
Standards  staff  for  evaluation.  Training  Program  Management  Officers  (TPMOs) 
should  have  a coordinating  and  consulting  function. 


(7)  During  the  conduct  of  the  training  program,  and  at  its  conclusion,  the 
Evaluation  and  Standards  staff  measures  the  trainee’s  ability  to  meet  training 
objectives  at  required  performance  levels. 

(8)  The  Evaluation  and  Standards  staff  reports  results  of  evaluating  training, 
and  field  performance  to  Academy,  SMS,  and  TR1. 

(9)  Identified  requirements  for  course  revision  are  transmitted  to  Academy. 

(10)  Academy  evaluates  course  revision  request  in  terms  of  the  Evaluation  and 

Standards  staff  reports  and  transmits  documented  request  for  change  to  TR1. 

Implementation 

The  improved  structure  can  be  established  by  cognizant  authorities  as  soon  as  they 
decide  that  it  meets  the  needs  of  the  FAA.  However,  the  information  which  is  required 
to  make  the  system  meaningful  must  be  developed  in  the  context  of  redeveloping  the 
curriculum  as  described  elsewhere.  Consequently,  the  major  costs  for  establishing  the 
system  rest  with  the  reorientation  of  the  present  course  structure. 

The  establishment  of  the  system  concurrent  with  the  revision  of  the  curriculum  is  a 
high  priority  item.  It  will  be  a major  tool  to  insure  that  the  training  product  is 
cost/effective. 
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Chapter  2 

ANF  MAINTENANCE  TRAINING  BRANCH-  SPECIFIC  PROBLEM  '. 
RECOMMENDATIONS,  AND  IMPLEMENTATION  PLANS 


There  are  fundamentally  two  kinds  of  recommendations  that  flow  Iron  n*  1 -i 
of  this  study.  The  first  are  programmatic,  high  priority,  and  of  general  »u-n 
cance,  and  have  already  been  discussed.  The  second  recommendation'  ar-  . if 
character  and  quality  of  the  instructional  system  as  it  was  observed  and  luo 
which  can  be  remedied  relatively  quickly.  Along  with  these,  other  re<  ommendau.M*  u 
will  take  more  time  and  effort,  and  some  which  should  await  the  < ondu<  t >•  . .r 
actions,  are  presented. 

The  form  of  the  recommendation  includes  a brief  statement  mtr  -n.  mg 
recommendation  in  the  context  of  the  findings  which  led  to  it  a staumen 
recommendation  itself;  and,  finally,  suggestions  on  how  the  recommcndati  . 
implemented. 


INSTRUCTIONAL  SYSTEM 


Visual  Aids.  From  a study  of  Instructor  Guides  and  other  cours*  material-  an 
direct  observation,  it  was  found  that  increased  utilization  of  audiovisual  aid 
benefit  the  instructional  process. 


Recommendations 


ud»  IN" 


(1)  Inform  instructors  of  availability  of  sources  for  additional  vim. a 
page  39) 

(2)  Provide  instructors  with  additional  training  in  the  use  of  audiovisual  aid- 

(3)  Increase  utilization  of  motion  pictures  and  filmstrips. 

Implementation 

(1)  Inform  instructors  of  sources  identified  now. 

(2)  Commence  training  instructors  now  in  visual  aids  course  at  Ac  ademy 

(3)  Upon  identification  of  relevant  motion  pictures  and  filmstrips  from 
cited,  or  others,  acquire  same  and  incorporate  in  training  program-  as  instru 
become  qualified. 


CONDUCT  OF  LABORATORY  EXERCISES 

It  was  found  that  laboratory  instructors  tended  to  be  limited  to  lal«>rators  . »•  r _ 
instructions  to  guide  the  students  through  their  work.  Increased  learning  ouicl 
expected  to  take  place  if  the  instructors  were  to  provide  greater  individualized  guidac  «■» 
to  students.  The  problem  of  providing  greater  guidance  was  aggravated  in  some  »r-  i 
the  fact  that  students  would  be  working  on  different  experiments  during  th* 
laboratory  period.  Since  the  several  experiments  pertain  to  theoretical  material  ’.n.g' 


several  previous  lecture  periods,  immediacy  of  application  of  knowledge  to  the 
experiment  could  not  be  achieved  in  all  cases. 

Troubleshooting  exercises  on  the  prime  equipment  should  be  as  realistic  as  possible. 
While  ease  of  insertion  and  removal  are  important  factors  in  selecting  troubles,  the 
troubles  should  realistically  represent  actual  field-experienced  difficulties. 

Some  laboratory  exercises  observed  were  actually  demonstrations.  As  performed, 
they  consumed  far  more  equipment  time  than  would  be  required  were  they  to  be  treated 
as  demonstrations.  Moreover,  the  prime  equipment  should  be  utilized  for  the  develop- 
ment of  a maintenance  skill. 

Prime  equipment  is  not  always  needed  for  teaching  a procedure.  Initial  skill  develop- 
ment and  learning  of  the  procedure  can  be  taught  on  a low-cost  mock-up.  Employing 
such  devices  would  have  the  effect  of  reducing  the  demand  for  the  prime  equipment  for 
teaching  maintenance  skills  which  require  it. 

From  a study  of  interviews  with  operating  personnel  and  other  data,  it  was  found 
that  preventive  maintenance  can  be  effectively  taught  on  site. 

Recommendations 

(4)  Require  instructors  to  provide  greater  individualized  guidance  to  students  during 
laboratory  exercises. 

(5)  Ensure  that  laboratories  occur  in  time  sequence  as  close  as  possible  to  the 
related  theory  presentation. 

(6)  Convert  laboratory  exercises  to  demonstrations  whenever  the  exercise  is  not 
devoted  to  development  of  a maintenance  skill. 

(7)  Select  troubleshooting  problems  representative  of  field  experience  for  insertion 
into  prime  equipment. 

(8)  Develop  and  use  low-cost  procedural  trainers  rather  than  the  prime  equipment 
for  teaching  procedural  tasks.  This  should  be  done  whenever  prime  equipment 
time  is  limited  due  to  heavy  scheduling. 

(9)  Use  on-site  equipment  to  teach  preventive  maintenance. 

Implementation 

(4)  This  could  be  done  by  administrative  order,  followed  by  supervisor  spot-checks. 

(5)  This  could  be  done  by  administrative  order.  To  keep  students  who  work  rapidly 
and  perform  the  experiments  well  in  advance  of  the  norm  profitably  employed, 
they  can  be  utilized  to  help  slower  students  in  a peer-instructional  method. 

(6)  Each  laboratory  exercise  that  uses  prime  equipment  should  be  examined  to 
identify  and  name  the  maintenance  skill  that  is  being  developed.  If  no  mainte- 
nance skill  is  being  developed,  convert  the  laboratory  into  a demonstration. 

(7)  Maintenance  data  should  be  collected  from  field  sites  to  determine  the  cause, 
frequency,  and  criticality  of  equipment  failures.  A representative  sample  of  these 
failures  can  then  be  selected  as  troubleshooting  training  problems.  The  effort  to 
collect  such  data  for  each  equipment  for  which  training  is  being  provided  can 
begin  immediately. 

(8)  Each  laboratory  exercise  should  be  reviewed  to  determine  whether  it  teaches 
procedural  tasks,  such  as  “turn  on,  operate.”  For  those  that  do,  prepare  scale 
drawings  to  be  used  in  place  of  the  equipment  for  this  training.  Work  on  making 
these  determinations  can  be  started  now. 

(9)  Instructor  Guides  should  be  examined  to  determine  whether  preventive  mainte- 
nance is  being  taught  in  resident  training.  When  detected,  the  content  should  be 
placed  in  QJT  materials,  and  removed  from  resident  training  materials.  Work  on 
this  can  commence  now 
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COURSE  MATERIALS 


Although  Instructor  Guides  are  required  to  contain  a statement  of  student  out- 
comes, there  is  no  requirement  that  Lesson  Plans  (which  are  prepared  by  the  instructor 
for  his  own  use  and  are  not  controlled  by  management)  contain  them,  unless  the  lesson 
plans  are  to  be  included  in  the  course  control  document  marked  as  “instructor  guides.”  If 
lesson  plans  are  not  required  to  contain  student  outcome  statements,  there  is  little 
likelihood  that  instructors  will  remember  what  they  are  and  inform  the  students.  There  is 
a requirement  that  Instructor  Guides  contain  “suggestions  how  student  outcomes  can  be 
achieved.”  Yet  there  is  little  evidence  to  show  that  these  suggestions  are  actually 
included. 

An  instructor  experienced  in  teaching  a course  will  probably  know  what  visuals  to 
select  for  a given  lesson.  However,  a new  instructor  would  not.  Neither  would  training 
management  officials  who  desired  to  review  the  appropriateness  or  quality  of  the  visual, 
since  their  identification  is  not  usually  specified  by  number  in  Instructor  Guides. 

Tests  using  multiple-choice  questions  are  very  popular  because  of  their  ease  of 
administration  and  scoring.  However,  such  tests,  except  in  rare  instances,  measure  a 
student’s  ability  to  recognize  the  right  answer,  rather  than  his  ability  to  perform. 

Training  materials,  such  as  student  guides,  Vugraphs,  and  laboratory  exercise  instruc- 
tions, contain  errors.  Some  of  these  errors  were  known  to  the  instructor  who  pointed 
them  out,  and  asked  students  to  make  the  necessary  corrections  when  they  could  (as  in 
student  guides).  Other  errors  were  detected  in  the  course  of  the  presentation.  In 
consideration  of  the  mass  of  training  materials  that  exists,  it  is  believed  that  the  problem 
is  not  severe;  but  because  it  appeared  during  the  observational  period  often  enough  to  be 
noticeable,  it  is  considered  to  be  worthy  of  attention. 

The  analysis  of  conent  within  a succession  of  courses  within  career  fields  detected 
evidence  of  content  duplication.  That  is,  topics  that  appeared  in  a first  course  also 
appeared  in  subsequent  courses  required  to  be  taken  for  certification  on  a given  equip- 
ment. Duplicated  content  is  clearly  wasteful  of  training  time  and  should  be  removed  if  no 
adequate  justification  can  be  made  for  its  retention.  (See  later  section  for  expanded 
discussion  of  content  duplication.)  However,  a determination  of  duplicated  content  for  all 
courses  should  be  made  after  career  progression  plans  have  been  established,  since  what 
now  might  appear  to  be  duplicated  content  in  the  context  of  present  training  paths  may 
not  be  after  the  training  paths  have  been  redefined. 

The  addition  of  a new  course  prior  to  redefining  the  present  training  paths  could 
also  result  in  the  duplication  of  course  material.  The  course  might  not  duplicate  material 
given  in  the  present  training  paths,  but  could  contain  duplication  after  the  new  training 
paths  were  developed. 

Recommendations 

(10)  Supervisors  should  see  that  lesson  plans  include  statements  of  student  out- 
comes. 

(11)  Require  visuals  to  be  explicitly  identified  by  number  or  other  identifier. 

(12)  Require  instructors  to  statp  purpose  of  lesson,  what  students  will  be  expected 
to  do  as  a consequence  of  the  instruction,  and  the  level  of  performance  they 
should  achieve  (when  the  later  information  becomes  available).  This  should  be 
done  at  the  beginning  of  the  lesson,  and  repeated  every  four  hours  if  the  lesson 
is  that  long. 

(13)  Enforce  the  requirement  that  “suggestions  on  how  student  outcomes  can  be 
achieved”  be  included  in  Instructor  Guides. 

(14)  Reduce  reliance  on  multiple-choice  questions. 

(15)  Require  training  materials  to  be  free  from  known  errors. 
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(16)  Remove  duplicated  topics  in  those  instances  where  they  were  detected  (as 
reported  in  Volume  I),  retaining  them  in  the  first  course  where  they  appeared 
if  analysis  of  job  requirements  justifies  their  presence. 

Implementation 

(10)  This  can  be  done  administratively,  and  implemented  now. 

(11)  This  can  be  done  by  clerical  personnel  with  assistance  from  instructors.  Instruc- 
tor Guide  masters  should  be  updated  so  that  future  reproductions  will  have  the 
necessary  data.  This  work  can  be  commenced  immediately. 

(12)  This  can  be  done  administratively,  implemented  now,  and  spot  checked  by 
supervisors. 

(13)  This  will  require  effort  from  instructors,  supervisors  and/or  curriculum  devel- 
opers to  write  the  necessary  additional  material  for  inclusion  in  the  Instructor 
Guides.  It  is  judged  to  be  a low  priority  item  for  a course  in  being,  but  a high 
priority  item  for  new  courses  (those  being  developed,  or  to  be  developed). 

(14)  This  will  require  a re-examination  of  the  content  currently  being  tested 
through  the  multiple-choice  question  method  to  construct  test  items  that  will 
require  the  student  to  demonstrate  his  ability  to  apply  the  knowledge.  Reason- 
able progress  in  the  conversion  of  multiple-choice  questions  should  be  visible 
within  a stated  time  frame,  for  example,  20%  within  three  months. 

(15)  Vugraphs  and  slides  containing  known  errors  should  be  corrected  immediately. 
Clerical  help  should  make  changes  on  student  guides  and  handouts  prior  to 
issuing  them  to  students  until  current  supply  is  exhausted. 

(16)  Topics  indicated  in  Volume  I as  duplicated  in  the  various  courses  should  not 
be  removed  until  instructors  teaching  the  topics  in  question  have  conferred  with  their 
counterparts  in  all  courses  in  the  related  training  path  containing  the  topic  to  determine 
that  duplication  actually  exists,  the  degree,  and  whether  the  requirements  of  the  job 
relative  to  the  first  training  program  where  it  appears  justifies  its  presence  there.  If 
conferees  determine  that  duplicated  topics  should  be  retained  in  courses  other  than  the 
first  where  it  appears,  they  should  set  forth  in  a report  the  reasons  for  their  determina- 
tion. Conferences  should  be  scheduled  immediately  for  this  purpose. 


TRAINING  EMPHASIS 

It  was  found  that  a sizable  portion  of  the  lecture  material  presented  in  resident 
instruction  in  equipment-specific  courses  was  devoted  to  descriptions  of  the  theory  of 
operation  and  circuits  in  their  normal  mode  of  operation.  Related  student  guides  for  such 
courses  often  contain  circuit  descriptions  at  a level  of  detail  comparable  to  that  provided 
by  the  instructor  in  class.  Since  students  are  permitted  to  retain  their  student  guides  for 
future  reference,  the  time  spent  by  instructors  in  describing  orally  the  content  of  the 
student  guides  appears  to  be  excessive. 

In  addition,  the  primary  emphasis  in  describing  circuit  theory  is  to  present  the 
normal  circuit  conditions.  Other  than  in  his  capacity  to  perform  preventive  maintenance, 
the  technician’s  real  usefulness  on  the  job  is  in  terms  of  how  well  and  how  fast  he  can 
restore  abnormal  circuit  conditions  to  normal.  Therefore,  in  order  to  enhance  this 
capability  it  is  reasonable  to  reduce  the  amount  of  time  spent  describing  normal 
operations,  since  it  is  covered  in  student  guides,  and  require  consideration  of  abnormal 
operation  in  the  context  of  teaching  normal  operation. 
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Recommendations 


(17)  Reduce  time  devoted  to  description  of  theory  and  operation  of  circuits  in 
courses  that  have  adequate  coverage  in  student  guides. 

(18)  Increase  emphasis  on  maintenance  in  resident  instruction  by  requiring  students 
to  practice  theoretical  troubleshooting  in  the  context  of  learning  circuit  theory. 

Implementation 

(17)  Instructors  and  supervisors  should  examine  student  guides  to  determine  which 
have  adequate  circuit  descriptions.  For  those  courses  which  do,  lecture  time 
devoted  to  circuit  descriptions  should  be  reduced.  A target  to  strive  for  would 
be  a reduction  of  as  much  as  50%,  depending  on  the  characteristics  of  a 
particular  course.  Instructor  Guides  and  lesson  plans  should  then  be  modified 
to  reflect  the  change  in  depth  of  coverage  in  the  lectures,  and  time  reduction. 

(18)  Requiring  students  to  practice  theoretical  troubleshooting  in  the  context  of 
learning  circuit  theory  will  require  rewriting  Instructor  Guides  and  lesson  plans 
to  include  problems  for  solution,  with  statements  of  symptoms  observed,  and 
condition  of  signals  resulting  from  the  hypothetical  out-of-tolerance  condition. 
Implementation  of  this  recommendation  should  be  deferred  for  those  courses 
likely  to  be  reorganized,  in  order  to  avoid  reworking  course  materials  more 
than  once.  The  maintenance  orientation  can  also  be  facilitated  through  drawing 
from  field  experience. 


ADMINISTRATIVE  AND  COURSE  CONTROL 

It  was  observed  that  training  outcomes  are  not  uniformly  specific.  To  have  utility, 
these  outcomes  must  be  phrased  in  terms  that  are  as  specific  as  practical. 

In  addition  to  the  purely  technical  data  required  for  course  generation,  curriculum 
developers  need  a knowledge  of  the  specific  character  of  the  job  requirements  and  the 
performance  standards  which  are  required  of  the  student  as  he  reports  for  duty  after 
training. 

Although  there  is  no  assurance  that  instructors  will  follow  their  lesson  plan  even  if 
they  have  one,  it  is  nonetheless  mandatory  that  the  lesson  plan  cover  the  topics  identified 
in  the  Instructor  Guide.  Unless  they  do,  there  is  even  less  likelihood  that  instructors  will 
be  teaching  the  planned  topics. 

Since  lesson  plans  an*  developed  independently  by  instructors,  there  is  no  method 
short  of  physical  inspection  and  comparison  with  the  Instructor  Guide  to  determine  that 
lesson  plans  contain  the  required  ingredients. 

Recommendations 

(19)  Remove  the  training  outcome  “assume  full  maintenance  responsibility,”  and 
similarly  worded  statements. 

(20)  Require  that  job  requirements  and  performance  standards  be  utilized  by  curric- 
ulum developers  in  constructing  courses.  Require  Instructor  Guides  to  contain 
specific  measurable  training  performance  standards.  Key  the  Instructor  Guides 
to  the  job  requirements  and  performance  standards  so  that  instructional  con- 
tent may  be  objectively  justified. 

(21)  Require  supervisors,  in  their  semi-annual  instructor  evaluations,  to  compare 
lesson  plans  with  Instructor  Guides  to  determine  completeness:  for  example, 
student  outcomes,  time  allocation  to  sub-topics,  visual  aid  identification,  and 
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performance  standards  when  they  become  available.  The  Evaluation  and 
Standards  staff  should  also  include  this  as  part  of  its  evaluation  actions. 

Implementation 

(19)  This  can  be  done  now.  These  statements  should  be  replaced  with  specific 
training  outcomes  which  define  the  maintenance  responsibility  to  be  assumed. 

(20)  This  will  have  to  be  deferred  until  job  requirements  and  performance  standards 
have  been  specified. 

(21)  This  will  require  instructors  to  add  the  material  presently  in  Instructor  Guides 
that  is  missing  from  lesson  plans,  and  to  determine  what  time  allocation  to 
place  on  sub-topics.  This  should  be  done  immediately.  Supervisors  should  plan 
to  make  the  comparisons  at  their  next  normally  scheduled  evaluation.  The 
Evaluation  and  Standards  staff  should  make  certain  this  factor  is  considered  in 
their  evaluations. 


ON  THE  JOB  TRAINING 

It  was  found  that  sector  management  could  benefit  from  additional  appreciation  of 
the  manpower  costs  associated  with  conducting  OJT  of  either  the  formal  or  informal 
types.  Moreover,  because  they  were  not  connected  into  the  communication  link  between 
the  Academy  and  the  trainee,  they  had  little  knowledge  of  the  state  of  training,  except 
when  a trainee  ran  into  difficulty.  The  quality  of  OJT  instructors  was  likely  to  be  highly 
variable  and  could  be  improved  through  training.  The  quality  of  the  training  itself  could 
be  improved  through  including  in  the  OJT  lesson  outlines  specific  points  or  facts  trainees 
are  supposed  to  be  able  to  do  or  state,  and  providing  the  trainee  the  opportunity  to 
practice  knowledge  immediately  after  learning  it.  (See  pp.  32-34) 

Recommendations 

(22)  Require  sector  managers  to  collect  man-hour  expenditures  from  all  sector 
personnel  for  their  participation  in  OJT. 

(23)  Include  sector  management  in  the  communications  link  so  that  they  may  keep 
abreast  of  the  state  of  training  of  OJT'  trainees. 

(24)  Add  specific  points  or  facts  trainees  are  supposed  to  be  able  to  do  or  to  state 
in  the  OJT  lesson  outlines. 

(25)  Provide  the  OJT  trainee  the  opportunity  to  practice  knowledges  immediately 
after  having  learned  them. 

(26)  Require  all  OJT  trainers  to  take  the  directed  study  program,  “On-the-Job 
Training  Techniques.” 

Implementation 

(22)  This  can  be  accomplished  immediately  by  administrative  order. 

(23)  This  can  be  accomplished  immediately  by  administrative  order,  to  include  the 
routing  of  communications  through  sector  management  office  where  training 
status  information  should  be  abstracted  and  recorded  for  review  and  analysts 
by  sector  management  personnel. 

(24)  Thus  will  require  a review  of  the  outlines  and  training  materials  to  determine 
what  the  points  are;  revision  to  incorporate  them  should  begin  now. 

(25)  Thus  is  a problpm  most  appropriately  solved  at  the  OJT  trainer  and  sector 
management  level,  although  guidance  can  be  provided  them  on  specific  topics 
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where  it  can  be  applied.  Operating  elements  should  be  directed  to  attend  to  it 
immediately. 

(26)  This  can  be  accomplished  immediately. 

LONG  TERM  RECOMMENDATIONS 

The  following  recommendations  are  interrelated  and  should  be  implemented  in  the 
context  of  re-engineering  training  programs. 

It  was  found  that  the  maintenance  concepts  studied  did  not  contain  sufficient  detail 
or  level  of  specificity  to  establish  measurable  training  objectives.  In  addition,  they  were 
not  written  so  as  to  reveal  the  aspects  of  equipment  design  and  utilization  that  impact 
upon  maintenance  requirements.  Maintenance  technician  performance  criteria  need  to  be 
defined  in  terms  of  what  is  needed  by  the  job  and  to  be  obtained  in  training. 

Since  performance  measures  are  unspecified,  the  quality  control  system  as  it  exists 
cannot  measure  the  training  program’s  effectiveness,  training  outcomes  are  deficient, 
instructors  have  no  standards  against  which  to  measure  student  achievement,  laboratories 
are  not  used  to  develop  measurable  performance  capabilities,  and  performance  levels  to 
be  achieved  in  OJT  are  not  known.  Re-engineering  of  the  training  programs  would 
include  the  specification  of  performance  standards  which  could  then  be  integrated  into 
the  course  materials  and  provide  the  substance  which  the  quality  control  program  could 
work  with. 

Recommendations 

(27)  Write  maintenance  concepts  so  as  to  delineate  what  aspects  of  equipment 
design  and  utilization  impact  upon  what  maintenance  will  be  required. 

(28)  Increase  detail  and  level  of  specificity  in  Maintenance  Concepts  so  as  to 
establish  measurable  training  objectives. 

(29)  Define  and  specify  training  outcomes  in  equipment-specific  terms  on  the  basis 
of  equipment  analysis,  utilization  analysis,  and  trainee  capability  to  include 
definition  of  performance  criteria  required  by  the  job  and  that  to  be  obtained 
in  training. 

(30)  Require  the  training  plan  Job  Function  Summaries  to  be  equipment-specific 
when  they  are  for  equipment-specific  training  programs. 

(31)  Establish  and  maintain  a specific  level  of  detail  and  time  allocation  within  and 
across  course  outlines.  The  course  outline  for  communication  equipment  may 
be  taken  as  a model. 

(32)  Develop  a quality  control  system  based  on  detailed  statement  of  training 
objectives  based  on  job  requirements,  proficiency  measures,  tests  which  require 
performance  rather  than  recognition  and  analyses  to  pinpoint  ineffective 
instruction.  Permit  flexibility  to  re-allocate  time  within  course  to  effect  quality 
control  determined  required  changes. 

(33)  Develop  diagnostic  tests  to  determine  what  part  of  the  training  program  needs 
improvement. 

(34)  Require  instructor  guides  to  contain  student  outcomes  stated  in  terms  meas- 
urable by  instructor  with  performance  standards  to  be  achieved  by  the 
students. 

(35)  Define  time  to  spend  on  each  major  topic  within  the  lesson  so  as  to  achieve 
greater  control  over  emphasis  of  presentation. 

(36)  Require  instructors  to  determine  that  outcomes  have  been  achieved  to  standard 
by  assessment  within  lesson.  Sample  student  achievement. 
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(37)  Design  laboratory  exercises  to  make  it  possible  for  students  to  achieve  mastery 
of  tasks  at  a level  required  of  them  upon  assumption  of  duties  on  site. 

(38)  Incorporate  performance  measures  and  standards  into  OJT  packages. 

(39)  Improve  integration  of  Directed  Study  and  OJT  I. 

Implementation 

(27)  & (28)  Work  should  begin  immediately  on  preparing  a guidebook  for  writing 
maintenance  concepts. 

(29  through  39)  Most  of  these  recommendations  depend  upon  the  identification  of 
training  objectives  based  on  job  requirements  and  performance  standards.  The 
most  effective  implementation  of  these  recommendations,  therefore,  should 
await  both  the  results  of  the  study  to  rank  order  the  course  in  their  relative 
priority  for  revision,  and  the  transfer  of  curriculum  engineering  technology  to 
ANF  personnel  which  is  recommended  to  be  performed  in  the  context  of 
re-engineering  five  selected  training  programs,  described  within  the  pro- 
grammatic recommendations  in  this  volume. 

There  is,  however,  no  particular  reason  to  defer  defining  time  allocations 
to  mqjor  topics  within  lesson  plans  so  as  to  achieve  greater  control  over 
emphasis  of  presentation  in  the  present  programs. 


CAREER  DEVELOPMENT 

There  is  no  centrally  managed  program  for  guiding  and  controlling  career  progression 
for  individual  technicians  from  point  of  entry  as  new  hirees  through  retirement.  As  a 
consequence,  it  is  difficult  to  project  personnel  and  training  requirements,  and  because 
there  is  no  career  progression,  training  is  costly  and  complex.  The  relative  freedom 
technicians  have  to  move  from  one  career  field  to  another  can  run  at  cross  purposes  with 
the  goal  of  maximizing  the  return  on  training  investment.  The  matter  of  establishing  a 
“core”  curriculum  is  related  both  to  career  progression  and  to  the  content  of  present 
training  programs.  Until  career  progression  patterns  have  been  established  which  are 
supportive  of  the  FAA’s  long-term  maintenance  work  force  goals,  and  until  the  present 
courses  have  been  redesigned  in  terms  of  job  requirements  and  performance  standards, 
the  development  of  a core  curriculum  would  be  premature.  (See  later  section  for 
additional  discussion  of  career  development  and  progression.) 

Recommendations 

(40)  Develop  plans  and  procedures  to  measure  personnel  career  progression  and  to 
project  personnel  and  training  requirements. 

(41)  Restrict  entry  prerequisites  for  those  courses  which  presently  permit  entry  of  a 
wide  mix  of  students  in  terms  of  their  training,  background,  and  experience. 

(42)  In  choosing  between  equally  qualified  technicians  to  cross  from  one  career 
field  to  another,  select  the  one  which  minimizes  the  loss  in  trained  investment. 

(43)  Defer  establishing  a core  curriculum  until  present  courses  have  been  refined  in 
the  light  of  training  requirements  and  performance  standards,  and  definition  of 
career  progression  patterns. 

Implementation 

(40)  A study  should  lie  initiated  at  the  earliest  practical  moment  to  define  the 
long-term  maintenance  work  force  goals  in  the  light  of  equipment  distributions, 


site  configurations,  and  known  anticipated  equipment  procurements,  and  to 
establish  training  paths  to  satisfy  the  goals  while  minimizing  loss  of  training 
investment. 

(41)  This  can  be  implemented  immediately  assuming  that  sufficient  numbers  of 
trainees  can  be  found  with  the  more  restricted  prerequisites  to  satisfy  the 
needs. 

(42)  This  can  be  implemented  immediately. 

(43)  Defer  implementation  of  “core”  curriculum. 


EQUIPMENT  UTILIZATION 

Even  though  pre-production  model  equipment  differs,  in  general,  from  production 
line  equipment,  there  are  definite  advantages  in  utilizing  it  for  certain  purposes.  In  the 
interests  of  acquiring  knowledge  of  an  equipment  in  the  pre-production  stage,  instructors 
can  be  sent  to  the  manufacturer’s  plant  for  training  on  the  prototype.  The  primary 
advantage  is  in  terms  of  the  time  required  to  prepare  instructional  materials.  Instructors 
so  trained  will  be  in  a better  position  to  construct  a training  program  for  the  production 
model,  although  it  may  differ  from  the  prototype,  in  a shorter  period  of  time  than  if 
they  were  required  to  wait  until  the  production  model  appeared. 

Recommendations 

(44)  Use  pre-production  model  equipment  for  training  course  development  per- 
sonnel at  contractor’s  site. 

Implementation 

(44)  Implement  as  required  and  appropriate. 


EQUIPMENT  RETIREMENT 

No  explicit  procedures  were  found  which  prescribe  a method  to  use  in  deciding 
when  equipment  should  be  retired.  Although  Paragraph  15,  Appendix  4 to  3006.6 A, 
Chg2  TRAINING  addresses  Training  Course  Termination,  the  three  program  standards 
state  that  “procedures  are  established”  to  help  determine  whether  a course  should  bq 
terminated,  not  if  and  when  to  retire  equipment.  It  is  quite  possible  to  find  a justifica- 
tion for  not  retiring  equipment  even  when  the  course  has  been  terminated.  Hence, 
training  course  termination  is  not  equivalent  to  equipment  retirement. 

Recommendation 

Develop  and  employ  equipment  retirement  policies  and  procedures. 

Implementation 

The  development  of  such  procedures  and  policies  would  not  have  a direct  impact  on 
the  improvement  of  ANF  training  It  would  have  an  indirect  effect  in  the  sense  that 
resources  currently  applied  to  the  maintenance  of  equipment  no  longer  required  could  be 
re-directed  to  more  pressing  requirements.  Consequently,  the  benefits  to  be  derived  must 
be  viewed  from  the  long  term  perspective. 

The  purpose  of  the  following  discussion  is  to  highlight  some  of  the  factors  which 
should  be  considered  in  the  development  of  such  policies  and  procedures. 
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Why  should  equipment  once  installed  at  the  Academy  ever  be  retired?  There  are  at 
least  two  reasons:  (a)  the  costs  to  maintain  the  particular  equipment  may  have  become 
excessive,  and  (b)  the  need  for  maintenance  training  on  the  equipment  may  have 
vanished. 

It  will  be  assumed  that  the  equipment  will  be  utilized  in  satisfying  attrition  training 
requirements.  After  some  period  of  utilization  has  elapsed,  the  costs  of  maintaining  an 
equipment  will  begin  to  rise  exponentially  because  the  failure  rate  will  rise  exponentially. 
If  a replacement  unit  were  to  be  purchased,  its  maintenance  costs  would  be  essentially 
flat  during  the  period  when  the  costs  of  the  first  equipment  were  rising.  If  the  difference 
between  the  maintenance  costs  for  the  two  equipment  were  computed,  it  would  be 
possible  to  determine  when  to  buy  the  second  equipment  so  that  the  savings  obtained  by 
not  maintaining  the  first  equipment  equalled  the  costs  of  procuring  the  second.  This, 
then,  would  be  the  economic  buy  time  for  the  second  equipment.  Naturally,  if  it  turned 
out  that  the  costs  to  procure  exceeded  the  savings  because  the  buy  point  exceeded  the 
anticipated  training  requirement  duration,  it  would  be  unwise  to  buy. 

The  second  factor  to  be  considered  is  the  requirement  for  the  equipment  in  training 
disregarding  its  state  of  repair.  Here,  for  those  instances  where  the  demand  for  training  is 
diminishing,  the  cost  of  maintaining  a training  program— that  is,  course  materials  and 
instructors  ready  to  perform— must  be  considered.  One  way  of  approaching  the  problem 
is  to  estimate  the  costs  of  converting  the  resident  training  program  to  directed  study  and 
OJT.  These  costs  could  then  be  divided  by  the  anticipated  student  load  over  the  life 
expectancy  of  the  equipment  to  obtain  a cost  per  student.  If  these  costs  were  less  than 
those  to  be  expended  in  maintaining  a resident  program  in  readiness  for  the  same  student 
load  over  the  same  time  period,  then  it  would  be  wise  to  discontinue  the  resident 
program. 


CAREER  PROGRESSION 

In  Volume  I of  this  report,  the  absence  of  clearly  defined  career  progression  patterns 
was  noted.  The  matter  of  content  duplicated  within  several  courses  and  that  which  is 
superfluous  to  the  needs  in  a given  training  course  is  related  to  the  career  progression 
problem. 

It  is  recommended  that  Career  Progression  Plans  be  developed  and  documented. 

Career  progression  practices  are  a very  important  determinant  of  the  overall  compo- 
sition of  the  FAA  Maintenance  Force.  They  have  a broad  interface  with  and  strong 
impact  upon  personnel  policies  and  the  ANF  training  mission.  Notwithstanding  the 
complexity  of  this  manpower  development  system,  there  remains  an  urgent  requirement 
for  management  to  place  some  structure  and  control  on  career  progression  of  its  airway 
facilities  maintenance  personnel. 

Full  benefits  from  the  training  investments  will  not  be  realized  until  definitive 
policies  have  been  established  to  describe  and  delineate  how  personnel  will  move  into  and 
through  certain  job  responsibilities  to  create  the  overall  maintenance  capability  imposed 
by  airway  facilities  equipment.  Firm  action  in  this  direction  would  also  create  the 
possibility  for  major  improvements  in  the  effectiveness  and  efficiency  of  ANF  training. 
Training  administrators  would  be  able  to  tailor  the  content  of  their  courses  more 
specifically  to  a predictable  training  input.  The  technicians  and  engineers,  working  with 
personnel  and  administration  representatives,  should  be  able  to  arrive  at  more  reliable  and 
valid  conclusions  concerning  their  future  careers. 

It  would  not  be  an  overwhelming  task  to  begin  by  defining  the  overall  maintenance 
capability  required  of  the  various  types  of  facilities  and  further  reducing  these  to  sets  of 
maintenance  capabilities  for  individual  personnel.  For  example,  what  equipment  system(s) 


would  a Radar  Technician  at  an  ARTCC  be  expected  to  maintain?  Given  a job  vacancy 
among  the  ARTCC  Radar  work  force,  what  is  the  most  efficient  sequence  of  training- 
work  experiences  to  satisfy  the  missing  maintenance  capability? 

With  the  NAS  program  picking  up  momentum,  a solid  effort  in  this  direction  should 
make  it  possible  for  all  parties  involved  to  develop  realistic  plans  to  meet  the  manpower 
requirements  of  this  program,  especially  for  those  instances  requiring  personnel  to  move 
across  career  fields.  Hopefully,  the  ability  to  make  longer-range  plans  concerning  person- 
nel training  needs  would  drastically  reduce  the  practice  of  waivering  prerequisite  require- 
ments, and  thereby  reduce  the  heterogeneity  of  the  training  input  at  ANF. 

The  certification  program  appears  to  be  the  single  most  important  element  in 
controlling  career  progression.  Due  to  its  criticality,  it  appears  that  efforts  should  be 
made  to  determine  if  the  two  different  parts  of  this  program  (theory  and  performance) 
are  assessing  the  maintenance  capability  required  and  if  they  are  measuring  the  same 
factors.  Since  the  written  examination  also  serves  as  a screening  or  selection  device  for 
certain  ANF  courses,  an  investigation  of  its  effectiveness  in  this  role  should  be  explored. 
It  is  possible  that  this  instrument,  used  to  serve  two  masters,  is  not  serving  either  one 
satisfactorily.  For  example,  one  would  suspect  a screening  instrument  based  upon  the 
content  in  the  Radar  Principles  courses  would  be  more  effective  than  the  system  concept 
oriented  R1  examination  as  a prerequisite  for  ASR-4,  5,  and  6 course.  There  is  a 
possibility  that  ANF  training  administrators  require  different  types  or  sets  of  data  for 
screening  the  training  input  where  the  individual  is  a new  hire  as  opposed  to  someone 
already  certified.  This,  too,  should  be  explored. 

TOPIC  DUPLICATION 

It  is  recommended  that  course  topic  duplication  be  re-evaluated  following  establish- 
ment of  career  progression  plans. 

Not  all  instances  of  duplicate  topic  between  courses  should  automatically  be  labeled 
as  inefficient  or  unnecessary.  Such  evaluations  should  be  reserved  for  situations  where  it 
can  be  clearly  demonstrated  that  a majority  of  students  in  one  course  received  instruction 
which  was  nearly  identical  to  instruction  that  they  had  received  in  a previous  course.  If 
this  instruction  is  found  to  be  repeated  for  a small  portion  of  the  students  and  can  be 
definitively  justified  in  terms  of  the  skills  and  knowledges  to  be  developed,  it  then  is  an 
instance  of  required  inefficiency.  The  key  word  here  is  required. 

The  essential  element  to  the  determination  of  what  is  required  is  a specific  state- 
ment describing  behaviorally  what  the  student,  will  be  able  to  do,  the  manner  in  which  he 
will  show  this  behavior,  and  a standard  of  accuracy,  speed,  or  precision  to  be  attained  as 
a result  of  the  instruction.  Furthermore,  these  instructional  objectives  should  be  directly 
related  to  specific  maintenance  tasks  found  on  the  job.  The  final  selection  of  topics  for  a 
given  course  involves  consideration  of  knowledges  and  skills  expected  to  be  present  in  the 
training  input.  Any  constructive  effort  by  course  developers  to  correct  or  prevent  topic 
duplication  is  not  likely  to  reap  full  benefits  unless  their  assessments  of  the  planned 
training  input  are  accurate.  This  is  not  likely  to  be  the  case  until  some  definitive  policies 
concerning  career  progression  are  established. 

The  ANF  resident  curricula  might  be  grouped  into  three  categories.  The  first  group 
would  contain  those  courses  aimed  at  developing  basic  theories,  principles,  and  knowl- 
edges of  basic  components,  their  operation,  and  applications.  This  group  would  include 
the  following  courses:  Solid  State  Devices,  Digital  Logic  Principles,  Electrical  Principles, 
Solid  State  Fundamentals,  and  parts  of  the  ILS/VOR  Principles  and  Communications 
Equipment  and  Rho  Theta  Principles.  The  second  group  would  contain  those  courses 
concerned  with  components  and  circuits  found  in  a family  of  equipment  such  as  Radar 
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Principles  A and  B,  Rho-Theta  Principles,  Data  Processing  Principles  Phase  II,  and  Air 
Conditioning.  The  third  group  would  include  all  of  the  equipment-specific  courses 
designed  to  prepare  someone  to  maintain  a specific  end  item  system,  such  as  BRITE-1, 
RTC-3,  and  APULS. 

Addressing  the  topic  duplication  question  in  terms  of  these  groups,  personnel  are 
likely  to  attend  more  equipment-specific  courses  than  those  of  other  groups.  For 
detecting  duplicate  topic  within  a given  career  field,  the  greatest  return  would  be 
obtained  by  examining  the  equipment-specific  courses,  especially  for  the  Radar  and  Data 
Processing  career  fields.  Relating  this  to  the  charts,  the  assumption  is  that  once  someone 
is  certified  on  a given  Radar  equipment  system,  the  training  progression  becomes  vertical 
in  most  instances.  This  would  not  appear  to  be  the  case  in  the  Navaids  career  field  where 
the  thrust  of  training  progression  could  be  horizontal  until  certification  on  three  different 
equipments  is  achieved.  In  attempting  to  identify  topic  duplication  between  the 
equipment-specific  group  of  courses,  one  should  examine  closely  :nstruction  devoted  to 
the  development  of  maintenance  skills  involving  the  operation  and  use  of  test  equipment, 
maintenance  techniques  or  procedures,  and  detailed  analysis  of  circuits  found  in  both 
equipment  systems. 

Any  attempt  to  identify  topic  duplication  resulting  from  career  progression  across 
career  fields  can  only  be  speculative  in  the  absence  of  any  information  of  typical 
patterns.  In  the  mjyority  of  cases  of  crossing  to  a new  career  field,  personnel  would  have 
to  attend  some  course  in  the  second  group -instruction  related  to  a family  of  equipment. 
For  a Radar  Technician  to  become  a Navaids  Technician  he  would  have  to  take  (in  most 
instances)  ILS/VOR  Principles  and  Rho-Theta  Principles.  Career  progression  in  the 
opposite  direction  would  mean  attending  the  Radar  Principles  course(s).  This  group  of 
courses  should  receive  a high  priority  in  detecting  content  duplication. 

Looking  at  the  Radar  and  Data  Processing  career  fields  on  the  chart,  it  is  apparent 
that  the  Solid  State  Devices  and  Digital  Logic  Principles  courses  are  present  in  most  of 
the  training  paths.  In  the  Radar  career  field  these  courses  are  followed  by  Radar 
Principles  A,  and  in  the  Data  Processing  area  they  precede  the  Data  Processing  Principles 
course.  A natural  assumption  would  be  that  the  Radar  Principles  and  Data  Processing 
Principles  courses  have  something  in  common  or,  if  not,  their  prerequisite  courses  contain 
instruction  which  can  be  justified  by  one  but  not  the  other. 

With  the  new  jobs  being  created  in  the  environmental  controls  system  area,  the 
requirement  for  training  experience  in  solid  state  and  digital  circuits  suggests  a closer  look 
at  the  Solid  State  Devices  and  Solid  State  Fundamental  courses.  This  is  based  upon  the 
assumption  that  sometime  in  the  future  large  numbers  of  electro-mechanics  will  begin  to 
cross  into  other  career  fields. 


POSSIBLE  MEANS  FOR  IMPROVING  COST  AND 
EFFECTIVENESS  OF  VARIOUS  TRAINING  METHODS 

The  objective  of  performing  this  task  was,  specifically,  to  arrive  at  recommendations. 
Consequently,  the  results  of  the  effort  are  reported  in  Volume  II  rather  than  in  Volume 
I.  Four  methods  are  discussed:  Resident  Training,  On-the-Job  Training,  Directed  Study, 
and  Programed  Instruction.  In  addition,  an  analysis  of  cost  factors  to  be  considered  in 
preparing  programed  instructional  materials  is  presented. 

For  the  most  part,  these  recommendations  can  be  instituted  independently  of  any 
general  changes  in  the  ANF  Branch’s  approach  to  training.  They  can  become  part  of  the 
daily  measures  taken  by  personnel  to  develop,  update,  and  improve  courses. 

During  the  course  of  this  effort,  numerous  possibilities  for  the  improvement  of  FAA 
electronic  maintenance  training  have  been  identified  and  considered.  These  possibilities 
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were  examined  in  the  context  of  the  individual  training  method  for  which  they  were 
found  applicable.  The  training  methods  included  Resident  Training,  OJT,  Directed  Study 
and  Programed  Instruction.  The  changes  recommended  are  discussed  below. 

A.  Resident  Training 

(1)  Increase  the  level  of  detail  and  specificity  of  the  Instructor’s  Guides. 

An  improvement  in  the  effectiveness  of  classroom  lectures  can  be  expected 
because  lecture  content  would  be  more  standardized  and  key  knowledges 
more  uniformly  emphasized  across  classes  and  instructors. 

(2)  Develop  and  include  specific  statements  of  student  outcomes  in  all  Instructor 
Guides  for  all  units  of  instruction. 

FAA  Handbook  No.  3000.6A,1  paragraph  404d(l),  provides  an  excellent 
definition  of  a properly  written  outcome.  The  availability  and  utilization 
of  more  specific  student  outcomes  by  instructors  would  increase  the 
effectiveness  of  training.  The  basic  psychological  concepts  which  would  be 
involved  and  require  the  presence  of  student  outcomes  are  defined  in  A 
Program  of  Instruction  for  Instructor  Training  prepared  by  the  Training 
Development  Division  of  the  FAA  Academy.  The  definitions  are  as 
follows:  “These  concepts  emphasize  that  instruction  is  more  meaningful 
and  effective  when  the  subject  matter  is  furnished  to  fulfill  an  identified 
need  to  improve  performance.  Effectiveness  is  increased  when  the  subject 
matter  is  presented  in  relation  to  the  situation  in  which  it  will  be  used 
rather  than  as  isolated  information.” 

FAA  Handbook  No.  3000. 6A  states  the  outcomes  “must  be  written  in 
measurable  terms  and  be  job  function  oriented.”  The  utilization  of  more 
specifically  defined  student  outcomes  by  instructors  to  (a)  identify  and 
emphasize  how  knowledges  to  be  developed  in  training  relate  and  are 
applied  to  specific  maintenance  activities,  and  (b)  identify  for  the  student 
the  performance  standards  he  is  to  achieve  during  a unit  of  training,  would 
make  substantial  improvements  in  the  effectiveness  of  the  training 
conducted. 

(3)  Develop  and  incorporate  diagnostic  tests  as  part  of  existing  training  quality 
control  system. 

When  instances  occur  where  significant  numbers  of  students  fail  to  achieve 
the  outcome  standards  established  for  some  unit  of  instruction,  effective 
corrective  action  is  dependent  upon  accurate  identification  of  the  cause 
for  the  occurrence.  Diagnostic  tests  can  provide  the  means  for  determining 
whether  the  knowledges  being  developed  in  class  are  appropriate  to  and 
required  by  the  behavior  defined  in  the  student  outcome.  On  the  other 
hand,  they  can  also  determine  whether  the  failure  is  attributable  to  the 
use  of  ineffective  training  media  or  methodology.  The  implementation  of 
diagnostic  testing  as  part  of  the  quality  control  system  would  result  in 
more  accurate  identification  of  the  causes  of  training  ineffectiveness  and 
the  corrective  actions  required. 

(4)  Change  emphasis  and  methodology  of  within-course  testing. 

An  improvement  in  the  effectiveness  of  training  should  be  expected  if 
relative  achievement  testing  was  de-emphasized  and  replaced  with  testing 
to  assess  the  extent  to  which  each  individual  student  attained  the 
prescribed  outcome.  Since  the  maintenance  requirements  of  an  equipment 

'Federal  Aviation  Administration.  Training.  FAA  Handbook  No.  3000.6A,  11  September  1969. 
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are  not  relative  to,  and  do  not  change  as  a function  of  individual  dif- 
ferences of  student  members  of  a class,  or  differences  between  successive 
classes,  the  required  level  of  achievement  for  a given  outcome  must  be 
absolute  for  each  student  and  independent  of  a classmate’s  performance 
and  that  of  previous  classes. 

The  reliance  on  multiple-choice  test  questions  should  be  decreased.  In 
those  instances  where  the  outcome  states  the  student  must  possess  a given 
knowledge,  a measurement  methodology,  which  requires  the  student  to 
recall  and  formulate  the  objective  response,  is  a more  valid  means  of 
testing  for  the  achievement  of  that  objective  than  one  which  requires  the 
student  to  recognize  the  knowledge  among  several  alternative  choices. 

This  change  would  cause  some  increase  in  test  administration  and  evalua- 
tion time.  However,  the  additional  costs  may  be  more  than  justified  by 
the  improvement  in  the  effectiveness  of  the  measurement  of  student 
outcome  achievement. 

(5)  Selection  of  more  appropriate  media  for  training. 

Observations  of  ANF-conducted  training  suggest  there  is  some  opportunity 
to  reduce  the  cost  of  training  for  certain  courses  by  selection  of  less 
expensive  media.  There  were  instances  of  laboratory  demonstrations 
conducted  on  the  prime  equipment  which  could  have  been  conducted  on  a 
mock-up  with  little  expected  loss  in  effectiveness.  Re-examination  of 
laboratory  demonstrations  presently  requiring  the  prime  equipment  to 
determine  the  possibility  of  using  other  media  could  provide  a basis  for 
reducing  training  costs. 

(6)  Increase  the  level  of  supervision  and  monitoring  of  student  performance  during 
laboratory  exercises. 

One  instructor  cannot  provide  an  effective  level  of  student  supervision  and 
monitoring  during  laboratory  exercises  involving  student  practice  on 
different  activities.  The  student’s  performance  needs  to  be  more  closely 
monitored  to  (a)  determine  that  he  is  practicing  the  task  assigned, 

(b)  detect  incorrect  responses  and  provide  corrective  guidance,  and 

(c)  assess  the  progress  being  made  toward  achieving  the  outcome.  The 
effectiveness  of  such  laboratory  exercises  would  be  greatly  improved  with 
the  addition  of  another  instructor. 

(7)  Increase  the  level  of  student  participation  during  lectures. 

The  effectiveness  of  the  instructor  in  presenting  information  to  the  stu- 
dents would  be  improved  by  increasing  the  level  of  his  interaction  with 
the  students. 

Learning  is  more  effective  when  the  student  is  active  than  when  he  is 
passive.  A more  concerted  effort  during  lectures  should  be  made  to 
determine  whether  the  student  comprehends  the  instruction  and  to  create 
situations  requiring  the  student  to  apply  that  which  he  has  learned.  To  be 
most  effective,  this  higher  level  of  student  participation  should  not  be 
restricted  to  either  the  beginning  or  the  end  of  the  lecture  but  should  be 
uniformly  present  throughout  the  duration  of  the  lecture. 

B.  On-the-Job  Training 

The  following  possibilities  for  improvement  of  the  cost  and  effectiveness  of  this 
method  of  training  are  restricted  to  that  form  of  OJT  found  in  integrated  programs;  that 
is,  OJT-I  and  OJT-II. 
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(1)  Increase  the  standardization  of  knowledges  to  be  learned. 

The  supervisor  in  his  role  as  the  trainer  is  a critical  factor  in  determining 
the  level  of  skills  and  knowledges  trainees  acquire  during  an  OJT  program 
It  is  logical  to  expect  these  personnel  in  this  role  to  vary’  considerably  m 
terms  of  technical  competence  and  as  a medium  of  instruction.  Such 
variability  can  be  expected  to  be  reflected  in  the  quality  of  the  training 
product— the  OJT  trainee.  The  OJT  manuals  are  excellent  examples  of 
efforts  to  compensate  for  trainer  individual  differences  and  to  assure  a 
more  uniform  product  of  known  quality. 

However,  there  is  some  possibility  for  greater  improvements  in  this  duv< 
tion.  In  those  instances  where  an  OJT  lesson  outline  contains,  for 
example,  an  objective,  such  as  “demonstrate  a basic  knowledge  of 
the  training  would  be  more  effective  if  the  trainer  were  provided  a list  of 
the  specific  points  or  facts  the  trainee  should  be  able  to  provide  The 
trainer  needs  more  specific  guidance  as  to  what  specific  pieces  of  informa 
tion  constitute  an  adequate  “basic  knowledge”  and  satisfy  the  lesson 
objective.  A very  conscientious  trainer,  being  uncertain  as  to  the  speeifu 
aspects  of  the  knowledge  involved,  would  probably  attempt  to  examine 
the  objective  and  establish  his  own  criteria  for  determining  if  the  objectiv* 
was  attained.  Thus  also  suggests  the  possibility  of  an  additional  benefit, 
that  is,  the  trainer,  having  been  provided  the  means  for  evaluation,  would 
have  to  spend  less  time  in  determining  whether  the  trainee  satisfied  th> 
lesson  objectives. 

(2)  Increase  the  utilization  of  the  “learning  by  doing”  principle  in  OJT-I. 

Based  on  study  of  the  OJT-I  manual  for  the  ATCB1-3  and  using  it  as  an 
example,  the  effectiveness  of  this  type  of  training  course  would  be 
improved  if  the  student,  were  required  to  practice  tasks  requiring  appln  a 
tion  of  knowledges  being  developed.  For  example,  shortly  after  comple 
tion  of  Lesson  8,  the  student  should  perform,  or  be  involved  in.  the 
performance  of  some  task  requiring  knowledge  of  safety  factors,  mainte 
nance  procedures,  name  and  location  of  major  equipment  units  and  their 
switches  and  controls,  and  other  knowledges  involved  in  this  set  of  lessons 

This  restructuring  of  the  OJT  program  would  also  result  in  making  the 
content  more  meaningful  to  the  student,  by  requiring  performance  of 
task(s)  which  clearly  demonstrate  a “need  to  know"  to  the  student,  and 
provide  him  with  the  opportunity  to  translate  knowledges  acquired  into 
performance.  Without  the  opportunity  to  practice  the  knowledges  to  be 
developed  by  this  program,  one  should  not  expect  the  student  to  retain 
much  of  that  to  which  he  was  exposed. 

In  discussions  with  field  personnel  concerning  types  of  content  most 
suitable  to  OJT,  preventive  maintenance  was  recommended.  It  would 
appear  that  preventive  maintenance  tasks  would  offer  an  opportunity  for 
practice  of  many  of  the  knowledges  forming  the  content  of  Lessons  1 
through  8. 

(3)  Provide  more  support  and  guidance  to  the  supervisor  in  his  role  as  a trainer. 

Field  personnel  reported  that,  one  of  the  principal  problems  with  OJT  was 
the  availability  of  personnel  who  were  competent  and  effective  as 
instructors.  Woolman1  reported  some  success  in  providing  relief  to  this 

1 M Woolman.  On  Site  Training  of  Guided  Missile  Operators,  HumRRO  Technical  Report  64. 

August  I960. 
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general  problem.  He  reports  development  of  an  on-site  training  method 
which  “gives  the  instructor-a  skilled  operator  but  usually  a novice  at 
teaching— an  easily  applied  technique  for  teaching  a procedure  to  his  two 
trainees.” 

The  method  included  standardized  training  procedures  and  supporting 
administrative  and  evaluation  materials.  Woolman’s  effort  in  this  area 
supports  the  possibility  of  a manual  for  the  OJT  trainer  which  would 
guide  and  direct  him  towards  a more  effective  role  as  an  instructor.  The 
effectiveness  of  this  manual  in  improving  the  quality  of  OJT  would 
depend  upon  the  extent  to  which  it  provided  the  trainer  guidance  and 
support  in  the  following  three  areas:  (a)  specification  of  student  behaviors 
to  be  observable  as  the  result  of  training;  (b)  instructional  media,  method- 
ology, or  techniques  to  be  employed;  (c)  definition  of  the  means  and 
standards  to  be  used  for  determining  whether  the  training  outcomes  were 
met. 

Another  possible  approach  for  improving  the  effectiveness  of  OJT  where 
the  basic  area  of  interest  is  the  trainer’s  role  as  an  instructor,  is  to  provide 
all  personnel  who  serve  as  OJT  trainers  with  OJT  Instructor  Training.  It 
would  appear  that  the  proposed  directed  study  program  on  “On-the-Job 
Training  Techniques”  could  serve  this  purpose.  However,  not  all  of  the 
content  proposed  for  this  program  would  be  required  to  provide  relief  to 
the  problems  discussed  here.  A shortened  version  of  this  program  which 
excludes,  for  example,  content  on  determining  training  needs  and  devel- 
oping training  objectives  would  be  more  appropriate  here.  The  present 
OJT-I  and  11  programs  prescribe  the  training  to  be  accomplished. 

Either  of  these  two  approaches  is  likely  to  produce  greater  improvements 
in  the  effectiveness  of  OJT-I  than  OJT-II.  The  reason  stems  from  the 
observation  that  OJT-1  is  concerned  with  the  more  cognitive  aspects  of  the 
job.  OJT-II,  on  the  other  hand,  is  aimed  more  at  the  development  of 
proficiency  in  tasks  for  which  a basic  performance  capability  was  previ- 
ously acquired  in  resident  training.  The  reason  also  rests  on  the  belief  that 
training  aimed  at  the  development  of  the  cognitive  components  of 
complex  tasks  places  greater  demands  on  the  pedagogical  skills  of  a trainer 
than  training  to  develop  speed  and  accuracy  in  performance,  as  found  in 
OJT-II. 

C.  Directed  Study 

(1)  Increase  the  use  of  programed  instruction. 

Excellent  models  of  the  application  of  programed  learning  principles  in  the 
printed  text  medium  can  be  found  in  both  resident  training  (No.  40115 
and  No.  40504)  and  directed  study  courses  (I/O  Equipment  for  Techni- 
cians). The  conversion  of  a study  manual  from  a narrative  format  (e.g., 
ATCBI-3)  to  “programed”  text  (e.g.,  I/O  Equipment)  would  be  expected 
to  increase  student  learning. 

However,  even  without  conversion  to  a programed  text,  other  possible 
means  of  improvement  exist.  Comparison  of  the  directed  study  manuals 
for  the  Diesel  Engine  Generator  and  ATCBI-3  resulted  in  the  opinion  that 
the  Diesel  Engine  Generator  ID  EG)  would  produce  greater  student 
learning,  since  the  DEG  manual  makes  more  effecthe  use  and  application 


of  certain  established  principles  of  learning.  These  principles  center  on 
three  aspects  of  the  learning  process— practice  of  the  knowledges,  knowl- 
edge of  results,  and  reinforcement  of  correct  responses.  On  a comparative 
basis,  the  DEG  manual  (a)  has  more  provisions  for  the  student  to  apply  or 
practice  what  he  has  learned,  (b)  provides  practice  more  frequently  and 
after  shorter  bodies  of  content,  and  (c)  has  provisions  for  telling  the 
student  whether  he  has  made  the  correct  response— that  is,  whether  he  has 
learned  the  material. 

Modification  of  the  more  purely  narrative  type  manuals,  such  as  for  the 
ATCBI-3,  to  include  those  features  identified  earlier  should  result  in  more 
effective  training.  The  addition  of  learning  objectives  should  further 
improve  the  effectiveness  of  the  DEG  manual. 

(2)  Improve  the  integration  of  Directed  Study  with  OJT-I. 

Examination  of  student  study  materials  from  both  types  of  courses 
suggests  the  possibility  for  improvement  in  both  courses  by  providing 
more  specific  information  to  the  trainee  and  trainer  on  how  the  courses 
are  to  be  conducted  concurrently.  A possible  analogy  of  the  relationship 
between  directed  study  and  OJT-I  is  the  classroom  lecture  and  laboratory 
exercise  in  resident  training.  This  relationship  appears  to  require  better 
definition,  coordination,  and  scheduling.  Consider  the  relationship  between 
ATCBI-3  directed  study  content  on  the  transmitter  site  and  Lesson  No.  5 
in  OJT-I  which  involves  a familiarization  tour  of  the  transmitter  site.  If  a 
student  completed  the  directed  study  lesson  two  or  three  days  before  the 
tour,  it  is  very  likely  he  would  have  forgotten  much  of  what  he  studied. 
Furthermore,  if  it  were  clearly  stated  to  him  what  specific  information  in 
the  directed  study  lesson  he  would  be  required  to  provide  during  the 
OJT-I  tour,  he  could  study  more  effectively. 

These  comments  can  be  applied  to  other  similar  situations  where  knowl- 
edges developed  in  directed  study  are  required  to  be  demonstrated  in  an 
OJT-I  lesson. 

There  is  some  evidence  to  support  an  observation  that  the  same  directed 
study  and  OJT-I  course  content  would  not  be  equally  effective  in  pre- 
paring a new  employee,  an  employee  changing  career  fields,  or  an 
employee  who  is  seeking  certification  on  a second  or  third  equipment 
within  the  same  career  field  for  a “smooth  transition”  into  resident 
training.  There  are  likely  to  be  pronounced  differences  in  the  skills  and 
knowledges  which  have  to  be  developed  during  directed  study  and  OJT-I 
in  order  to  assure  smooth  transition  into  resident  training  by  a new  hire, 
compared  to  someone  already  certified  on  equipment  similar  to  that  to  be 
taught  in  the  resident  course.  This  would  suggest  courses  with  different 
content  and  possibly  employing  different  training  methods  and/or  media. 

D.  Programed  Instruction 

Programed  instruction  was  examined  as  an  alternative  technology  that  might  be  used 
in  one  or  more  of  the  10  courses  studied.  Purely  from  the  standpoint  of  effectiveness,  it 
can  be  considered  as  a method  that  could  be  employed  in  greater  amounts  in  ANF 
training.  However,  its  costs  can  be  too  large  in  relationship  to  the  improvements  that  may 
be  affected. 

The  SCT  format  used  in  Course  No.  40115  can  be  improved  as  a programed 
instruction  (PI)  course.  However,  apart  from  the  question  of  programed  instruction  (PI) 
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being  a wise  choice  for  implementation  in  this  course,  one  feature  of  it  does  make  the 
course  superior  to  most  that  were  examined:  namely,  the  effort  to  place  the  material  in  a 
programed  instruction  (PI)  format  resulted  in  a greater  specification  of  what  the  student 
is  to  learn,  and  a strong  basic  organization  of  the  materials.  Therefore,  the  strength  of 
this  course  is  in  the  direction,  specification,  and  organization  of  course  content  that 
exceed  most  courses  that  were  examined. 

In  addition,  however,  the  use  o'  programed  instruction  in  this  course  is  a good 
choice  because  the  content  of  the  course  is  stable,  that  is,  it  is  unlikely  to  be  changed 
considerably  because  it  considers  principles  rather  than  material  on  a specific  piece  of 
equipment. 

Generally,  courses  that  are  likely  to  have  stable  content  are  more  suitable  for  the 
use  of  programed  instruction  than  those  that  are  subject  to  frequent  or  extensive  revision, 
because  they  will  not  require  costly  revision. 

Because  it  is  more  difficult  and  expensive  to  revise  programed  materials  than  those 
presented  by  a live  instructor,  programed  instruction  is  more  suitable  for  relatively  stable 
content  areas  than  for  those  areas  likely  to  experience  rapid  change.  Depending  upon  its 
form,  programed  instruction  can  provide  individualization  of  instruction  to  a degree  that 
would  be  excessively  time-consuming  in  a resident  instructional  situation.  And,  programed 
instruction,  regardless  of  its  form,  permits  the  individual  to  proceed  at  a pace  he  finds 
most  comfortable— obviously  impossible  to  do  in  a resident  training  situation. 

The  initial  cost  to  prepare  a program  is  generally  higher  than  that  to  prepare  a 
course  taught  by  a live  instructor.  This  is  because  the  program  is  constructed  with  an 
initially  high  capability  to  teach  the  subject  matter.  In  a resident  training  program,  the 
course  can  be  revised  and  tailored  by  the  instructors  almost  as  it  is  being  taught  In  a 
resident  program  the  costs  continue  to  occur,  due  to  the  presence  of  the  instructor,  while 
in  programed  instruction  the  costs  over  the  long  run  tend  to  be  those  of  maintenance  of 
the  program,  which  for  very  stable  subject  matter  would  be  minimal,  instructor  costs 
having  been  eliminated. 

Because  of  the  careful  analysis  which  precedes  the  construction  of  a programed 
course,  such  courses  tend  to  be  more  effective  than  those  taught  by  an  instructor.  There  is 
no  reason  why  the  same  careful  analysis  should  not  precede  the  constructing  of  a resident 
course.  If  this  were  done,  the  courses  might  be  almost  equally  as  effective  and  certainly 
much  less  expensive.  The  choice  as  to  which  method  to  use,  then,  would  depend  upon 
the  relative  costs,  and  the  consideration  of  factors  which  assign  values  to  either  method 
in  terms  of  advantages  and  disadvantages. 

The  following  is  a cost  analysis  of  a hypothetical  course  which  might  be  put  into  a 
programed  format. 

Example 

Assumptions: 

(1)  The  course  has  a stable  content,  not  likely  to  change  in  a five-year  period. 

(2)  The  course  presently  exists  as  a resident  course  with  50%  laboratories,  50% 
lectures. 

(3)  Lectures  are  presently  taught  by  one  instructor. 

(4)  Because  it  is  not  feasible  to  send  lab  kits  to  the  field,  students  will  attend 
resident  instruction  for  the  laboratory  portion  of  the  programed  course. 

(5)  The  program  will  be  constructed  by  Academy  instructors. 

(6)  The  rate  of  program  production  will  be  100  hours  of  instructor  effort  to 
prepare  one  hour  of  fully  tested  programed  material. 
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General  formulation: 


Let 


R 

h 

r 

N 

s 

CP 

P 

L 

Si 

Ss 

St 


= Instructor  salary,  $/hour 

= Number  of  course  hours  to  be  converted  into  programed  instruction  (PI) 
= Instructors  programming  rate  (hours  required  to  write  fully  tested 
hour  of  PI) 

= Number  of  times  course  presented  in  a year 
= Number  of  students  in  class 
= Cost  of  programming  (instructor’s  effort) 

= Per  diem  rate  $/day 
= Life  expectancy  of  program,  years 
= Savings  on  Instructor  salary  over  L 
= Savings  on  students  per  diem  over  L 
= S;  & Ss  (total  savings) 


Instructor  cost  of  programming: 

Cp  = R x r x h 

Savings: 

Instructor  salary:  Sj  = hxNxRxL 

Per  Diem : Ss  = -^-xpxsxNxL 

Total:  St  = hNL  (R  + -Hi) 
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To  make  the  example  as  explicit  as  possible,  some  further  assumptions  will  be  made: 

(1)  Present  course  length:  80  hours 

(2)  Amount  to  be  programmed:  40  hours 

Since  it  has  been  assumed  that  it  will  require  100  hours  to  present  one  hour  of 
instruction,  to  program  40  hours  will  require  4000  hours  of  programming  time.  The 
present  hourly  rate  of  a GS-12  is  approximately  $7.50.  Multiplying  the  hourly  rate  by 
the  number  of  hours  required  gives  the  cost  of  preparing  the  programed  material  in  terms 
of  instructors’  salaries. 

R x H = Cp  or  Cp  = R x r x h 

$7.50  x 4000  = $30,000 

Cp  = $30,000  Cp  = $7.50  x 100  x 40 

Cp  = $30,000 

It  is  assumed  that  students  will  still  come  to  the  Academy  to  perform  the  laboratory 
exercises.  Therefore,  travel  costs  will  be  the  same  in  both  the  resident  mode  and  the  PI 
mode.  However,  they  will  be  present  for  only  40  hours  instead  of  80  hours  of  instruc- 
tion, and  no  instructor  will  be  required  for  the  programed  portion  of  the  course.  Thus, 
there  are  savings  in  both  per  diem  for  students,  and  in  the  cost  of  instructor  salary.  How 
much  these  savings  amount  to  depends  upon  the  size  of  the  class  and  the  number  of 
repetitions  during  a year. 

Assume  16  classes  per  year,  12  students  per  class 

Instructor  salary  savings: 

S,  = hxNxRxL 

Sj  = 40  hours  x 16  classes  = $7.50  per  hour  x 5 years  = $24,000 
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Student  per  diem  savings: 

Ss  = -^-xpxSxNxL 
s 8 

40 

— days  x $16/day  x 12  students  x 16  classes  x 5 years  = $76,875 

O 


Total  Savings: 

Instructor  salary  $24,000 
76,875 

$100,875  over  5 years  or  $20,175  per  year. 


Time  to  amortize  instructor  cost  to  prepare  program: 


30,000 

20,175 


1.48  years. 


As  indicated,  over  the  five-year  assumed  useful  life  of  the  program,  the  savings  in 
instructor  and  per  diem  costs  would  amount  to:  $100,875. 

This  figure  is  probably  low  for  it  does  not  account  for  pay  increases  over  the 
five-year  period,  or  increases  in  per  diem  rates. 

Taking  it  as  a conservative  estimate  of  the  savings  due  to  instructor  salaries  and  per 
diem  expense,  the  figure  may  be  considered  to  be  the  break-even  point  for  a program 
which  has  a five-year  life  expectancy.  On  that  basis,  one  may  permit  the  cost  to  prepare 
such  a program  to  approach  the  $100,000  mark,  rather  than  the  $30,000  initially 
assumed. 

There  are,  of  course,  additional  costs  in  preparing  the  materials  than  the  cost  of 
programmers— cost  of  art  work,  reproduction  costs,  mailing  costs,  and  so  forth— but  these 
costs  would  have  to  be  almost  double  the  programmer  costs  before  the  break-even  point 
was  reached  in  the  example. 

As  a specific  example,  consider  Course  No.  40115,  Solid  State  Fundamentals  for 
Electromechanics.  The  ANF  Training  Quota  Assignments  for  FY  72  indicates  that  there 
are  14  classes  programmed,  with  an  average  of  16  students  per  class.  It  is  a two-week 
program  with  40  hours  devoted  to  laboratory  exercises.  Although  the  text  used  in  the 
course  is  a combination  of  narrative  and  “programed”  types,  the  self-instructional  capa- 
bility of  the  program  could  be  improved  either  by  reducing  the  size  of  “jumps”  between 
successive  frames,  or  by  inclusion  of  remedial  sub-routines.  In  either  event,  the  programed 
text  would  require  revision  and  expansion.  Were  this  to  be  done,  students  could  then  be 
brought  to  the  Academy  to  accomplish  the  laboratory  phase  of  the  program,  thus 
reducing  the  total  resident  time  to  40  hours. 

Applying  the  formulas  previously  developed,  the  following  results  are  obtained: 

Cp  = R x r x h 

= $7.50  x 100  x 40  = $30,000 

This  figure  is  undoubtedly  high  since  the  basic  structure  of  the  program  already  exists. 
For  this  reason,  25  hours  per  hour  of  instruction  is  probably  a better  estimate.  Thus: 


Cp  = $7.50  r 25  x 40  = $7,500 
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With  14  classes  per  year,  and  16  students  per  class,  the  savings  would  be  calculated 
as  follows: 

Sj  = hxNxRxL 

Si  = 40  x 14  x $7.50  x 5 = $21,000 

i , = — xpxsxNxL 
J 8 

5 x 16  x 16  x 14  x 5 = $89,600 
St  = Sj  + Ss  = $110,600 
or  $22,120  per  year. 

As  a consequence,  the  cost  of  revising  the  program  could  be  amortized  over  the  first 
year  of  its  operation  through  the  saving  realized  from  reducing  the  requirement  of  the 
live  instructor,  and  cutting  the  resident  portion  to  40  hours  of  laboratory  exercises. 

In  conclusion,  it  can  be  seen  that  for  the  assumption  made,  the  costs  of  preparing 
the  program  can  approach  the  saving  to  be  realized  over  the  life  expectancy  of  the 
program.  These  same  formulae  can  be  applied  to  any  course  material  that  becomes  a 
candidate  for  reconstruction  in  the  fl  format. 


AUDIO  VISUAL  SOURCES 

It  is  recommended  that  audio-visual  materials  be  employed  more  frequently. 

Motion  picture  films  are  particularly  useful  in  teaching  topics  that  require  visualiza- 
tion of  the  interaction  of  nearly  simultaneously  occurring  events,  movement,  and  flow. 
Verbal  presentations  and  the  use  of  chalkboard  and  Vugraphs  are  not  nearly  so  effective. 

Increased  usage  of  visuals  will  reduce  training  time  in  those  situations  where  they 
may  be  substituted  for  instructor-created  chalkboard  drawings.  So  little  use  is  made  of 
motion  picture  films  in  the  10  courses  that  it  is  possible  course  personnel  are  unaware  of 
certain  sources  they  may  turn  to. 

The  central  sales  point  for  most  completed  U.S.  Government  AV  items  is: 

National  Audio-Visual  Center 
General  Services  Administration 
Washington,  D.C.  20409 
It  publishes  a catalog  of  U.S.  Government  Filins. 

U.S.  Army  organizations  which  are  authorized  to  stock  and  to  lend  official  Army 
films,  and  so  forth,  are  Audio-Visual  Support  Centers.  Probably  the  closest  one  to 
Oklahoma  City  is: 

Audio-Visual  Support  Center 
Office  of  the  Division  Engineer 
Department  of  the  Army 
1114  Commerce  Street 
Dallas,  Texas  75202 

The  basic  listing  of  Army  films  and  other  AV  materials  is  DA  Pamphlet  108-1  (dated 
September  1966)  together  with  Change  1.  In  addition,  AVSCs  publish  a regular  news- 
letter announcing  new  films  and  other  services. 

Perusal  of  the  catalogs  from  these  sources  should  reveal  titles  of  potential  interest, 
especially  in  the  fundamentals  training  areas.  Copies  of  films  could  be  obtained  and 
reviewed  for  pertinency  and  timeliness  and  purchased,  if  desired. 
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INSTRUCTOR  PRODUCTIVITY  MEASURES 


It  is  recommended  that  effective  measures  of  instructor  productivity  be  developed. 

It  is  not  logical  to  expect  that  in  the  development  of  materials  for  a new  training 
program  the  ratio  of  hours  of  development  to  lesson  hours  to  be  developed  is  necessarily 
a fixed  ratio.  Allocation  of  time  for  such  activities  would  have  to  take  into  account 
training  objectives  and  standards  of  performance  to  be  achieved  by  trainees,  the  com- 
plexity of  the  equipment  system,  the  similarity  of  that  system  to  one  for  which  a 
training  program  had  already  been  prepared,  the  prior  experience  of  course  developers  in 
general,  and  in  particular,  with  respect  to  developing  materials  for  like  systems,  and 
historical  data  relevant  to  the  man-hour  expenditure  for  like  purposes  under  similar 
conditions  in  the  past.  Development  activities  required  for  equipment  which  was 
essentially  new  and  like  no  former  equipment,  and  which  had  to  be  prepared  by  course 
developers  who  had  had  no  prior  experience  in  preparing  course  materials  would 
necessarily  consume  a greater  period  of  developmental  time.  In  any  event,  the  most 
profitable  activity  to  undertake  in  this  current  instance  is  for  ANF  to  begin  collecting  the 
data  base  from  which  projections  can  rationally  be  made. 

Table  2 shows  how  to  assign  relative  time  weights  to  the  preparation  of  instructional 
materials.  It  takes  into  account  characteristics  of  both  the  equipment  for  which  training 
materials  are  to  be  prepared  and  personnel  characteristics  of  the  instructors.  The  system 
that  can  be  employed  to  gauge  productivity  is  then  explained  in  detail. 

Along  the  left  margin  of  the  table  is  a listing  of  a number  of  possible  products 
instructors  may  be  required  to  produce.  The  headings  classify  the  activities  in  terms  of 
equipment  and  personnel  characteristics. 

The  first  major  split  occurs  on  the  basis  of  whether  the  products  to  be  prepared  are 
for  a revision  of  an  existing  course,  or  whether  they  are  to  be  prepared  for  a new  course. 
Course  materials  class:  original  or  revision. 

The  second  split  occurs  on  the  basis  of  whether  the  equipment  is  complex  or  simple. 
One  way  of  making  this  decision  is  to  consider  the  relative  number  of  electronic 
components  or  stages  that  comprise  the  equipment,  and  the  presence  or  absence  of 
multiple  modes  of  configuration. 

The  next  split  is  on  the  similarity  of  the  equipment  to  existing  equipments.  This 
classification  is  particularly  important  with  respect  to  the  construction  of  original 
materials,  since  if  the  new  equipment  is  similar  to  an  existing  equipment,  it  is  possible  to 
take  advantage  of  some  materials  already  in  existence  to  guide  the  development  of  new 
materials,  and  instructors  may  already  have  much  of  the  knowledge  required  to  construct 
the  new  materials  because  of  their  familiarity  with  the  old. 

The  next  split  is  on  the  basis  of  the  writer’s  familiarity  with  the  equipment.  The 
writer  may  be  familiar  with  the  equipment  because  he  attended  a key  personnel  training 
program  conducted  by  the  manufacturer,  or  he  may  have  maintained  the  equipment  in 
another  job.  Since  it  is  possible  to  assign  writers  who  have  varying  levels  of  familiarity 
with  the  equipment,  some  weighting  should  be  given  for  this  factor. 

The  products  listed  in  the  left  margin  are  representative  of  the  materials  to  be 
produced  and  are  by  no  means  comprehensive.  The  numbers  in  the  body  of  the  table 
represent  the  relative  weighting  to  apply  to  the  production  of  the  materials  as  a function 
of  equipment  and  personnel  factors. 

For  example:  Under  Instructor  Guides,  lecture,  the  number  “1”  appears  in  the 
situation  where  a course  is  to  be  revised,  it  is  a simple  equipment,  like  others  being 
taught,  and  the  writer  is  known  to  be  familiar  with  the  equipment.  A “1”  also  appears 
when  the  equipment  is  unlike  other  equipments.  This  merely  means  that  since  the 
materials  are  to  be  revised  for  a simple  equipment  and  the  writer  is  familiar  with  the 
equipment,  similarity  of  the  equipment  to  others  has  no  effect  on  the  relative  time 
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required  to  write  lecture  Instructor  Guides.  The  number  “1”  is  the  multiplying  factor  to 
apply  to  the  time  it  takes  to  write  an  Instructor  Guide.  If  it  takes  one  hour  to  revise  an 
Instructor  Guide  for  a simple  equipment  which  is  like  other  equipments  by  an  instructor 
who  is  familiar  with  the  equipment,  it  will  take  two  hours  to  do  if  he  is  unfamiliar  with 
the  equipment  It  would  take  three  hours  to  do  if  he  was  unfamiliar  with  the  equipment 
and  there  were  no  similar  equipments. 

The  highest  weighting  is  assigned  to  the  writing  of  Instructor  Guides  in  the  situa- 
tions where  the  equipment  is  complex  and  unlike  no  other,  and  the  instructor  is 
unfamiliar  with  the  equipment. 

The  weightings  for  laboratory  Instructor  Guides  are  somewhat  higher  (about  20%) 
because  of  the  assumed  effort  required  to  try  out  the  materials  on  the  equipment. 

M Lesson  Plans  were  given  the  same  weighting  as  Instructor  Guides  since  although  the 
structure  and  sequence  of  materials  is  prescribed  in  the  Instructor  Guides,  the  details  of 
what  to  present  must  be  developed  for  the  Lesson  Plans. 

Student  Guides  were  arbitrarily  assigned  a weight  five  times  that  of  Instructor 
Guides.  The  reason  for  this  is  that  Student  Guides  are  texts  containing  pictorials  as  well 
as  verbal  matter. 

Visual  Aids  were  assigned  one-tenth  the  weight  of  Instructor  Guides,  and  Handout 
Materials  assigned  two-tenths  the  weight  of  Instructor  Guides  because  they  will  probably 
contain  more  text  than  visuals. 

As  can  be  seen,  the  numbers  in  the  body  of  the  table  have  a logical  relationship.  No 
claim  can  be  made  for  either  the  precision  of  the  absolute  number  or  the  relative  ratios. 
The  object  of  the  presentation  is  to  demonstrate  that  certain  factors  have  an  influence  on 
the  productivity  of  instructional  personnel  in  creating  certain  course  materials  and  that  a 
logical  framework  can  be  created  for  weighting  the  efforts. 

The  numbers,  however,  could  be  used  as  a first  approximation  and  modified 
appropriately  as  data  were  developed  in  the  process  of  revising  existing  course  materials 
and  in  developing  new  course  materials. 

In  order  to  implement  this  scheme,  ANF  should  classify  the  equipment-specific 
training  programs  in  the  manner  indicated,  determine  the  relative  familiarity  of  the 
instructor  assigned  to  rewrite  materials  and  collect  the  number  of  hours  required  for  the 
instructor  to  perform  the  work.  As  the  data  are  collected  and  summarized,  the  relative 
weightings  which  actually  exist  should  emerge.  When  sufficient  data  have  been  collected 
it  should  then  be  possible  both  to  evaluate  an  individual’s  productivity  against  the  norm 
for  that  kind  of  work,  and  to  project  manpower  requirements  based  on  known  average 
expenditures. 
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